Use of eye tracking to adjust region-of-interest (ROI) for compressing images for transmission

ABSTRACT

Gaze tracking data may representing a user&#39;s gaze with respect to one or more images transmitted to a user are analyzed to determine one or more regions of interest. 
     The one or more transmitted images are selectively compressed so that fewer bits are needed to transmit data for portions of an image outside the one or more regions interest than for portions of the image within the one or more regions of interest.

CLAIM OF PRIORITY

This application is a continuation of U.S. patent application Ser. No.15/087,471 filed Mar. 31, 2016, the entire contents of which areincorporated herein by reference.

FIELD OF THE DISCLOSURE

Aspects of the present disclosure are related to digital graphics. Inparticular, the present disclosure is related to varying thequantization within compressed image to maximize compression andminimize perceptible artifacts presented to a user.

BACKGROUND

Graphical display devices having a wide field of view (FOV) have beendeveloped. Such devices include head mounted display (HMD) devices. Inan HMD device, a small display device is worn on a user's head. Thedisplay device has a display optic in front of one eye (monocular HMD)or each eye (binocular HMD). An HMD device typically includes sensorsthat can sense the orientation of the device and change the scene shownby the display optics as the user's head moves. Conventionally, moststages of rendering scenes for wide FOV displays are performed by planarrendering where all parts of the screen have the same number of pixelsper unit area.

However, rendering for virtual reality (VR) programs, which is oftenperformed in conjunction with HMD devices, requires a higher frame ratethan conventional flat screen rendering to prevent a user fromexperiencing motion sickness. HMD for VR has optical systems to showrendered scenes in wide FOV for immersive experiences. While the screenarea around a primary gaze point (sometimes called the foveal region)requires high resolution, the areas outside the primary gaze point areobserved only by the peripheral vision and can therefore be rendered ata lower resolution, or may contain less detail. Such rendering issometimes referred to as foveated rendering.

Research has been performed that seeks to apply foveated rendering atthe pixel level by selectively adjusting the pixel resolution fordifferent parts of the screen. See co-pending U.S. patent applicationSer. No. 14/246,066, to Mark Evan Cerny, filed Apr. 5, 2014, which isincorporated herein by reference. Furthermore, the foveated renderingconcept may be applied at earlier stages of a graphics processingpipeline, such as the geometry level, e.g., by adjusting thetessellation of computer generated objects for different parts of thescreen on which they are displayed. See co-pending U.S. patentapplication Ser. No. 14/927,157 to Jun Murakawa et al. filed Oct. 29,2015, which is incorporated herein by reference. These approaches, andothers, can reduce the computational load on graphics processinghardware by concentrating computational resources on rendering moreimportant parts of an image on a display.

It is within this context that the present disclosure arises.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1B are schematic diagrams illustrating gaze tracking within thecontext of aspects of the present disclosure.

FIG. 2 is a flow diagram depicting a method according to aspects of thepresent disclosure.

FIG. 3 is a block diagram depicting a system according to aspects of thepresent disclosure.

FIG. 4A is a simplified diagram illustrating an example of normaltessellation performed in accordance with the prior art.

FIG. 4B is a simplified diagram illustrating an example of foveatedtessellation in accordance with aspects of the present disclosure.

FIGS. 5A-5B are flow diagrams depicting a graphics processing methodsaccording to an aspect of the present disclosure.

FIG. 6A is a schematic diagram of a screen space illustrating an exampleof a region of interest in accordance with aspects of the presentdisclosure.

FIGS. 6B-6D are graphs depicting examples of transitions in vertexdensity over a screen space in accordance with aspects of the presentdisclosure.

FIG. 7 is a block diagram of a graphics processing system in accordancewith aspects of the present disclosure.

FIG. 8 is a block diagram of a graphics processing pipeline that may beimplemented, e.g., by the system of FIG. 7 in accordance with aspects ofthe present disclosure.

FIGS. 9A-9H are schematic diagrams illustrating examples of the use ofeye gaze and face tracking in conjunction with embodiments of thepresent invention.

FIG. 10A-10D are schematic diagrams illustrating facial orientationcharacteristic tracking setups according to aspects of the presentdisclosure.

FIG. 10E is a schematic diagram illustrating a portable device that canutilize facial orientation tracking according to an aspect of thepresent disclosure.

DETAILED DESCRIPTION

Although the following detailed description contains many specificdetails for the purposes of illustration, anyone of ordinary skill inthe art will appreciate that many variations and alterations to thefollowing details are within the scope of the invention. Accordingly,the illustrative implementations of the present disclosure describedbelow are set forth without any loss of generality to, and withoutimposing limitations upon, the claimed invention.

Introduction

Eye gaze tracking has use in a wide range of applications, includingmedical research, automobile technology, computer entertainment andvideo game programs, control input devices, augmented reality glasses,and more. There are a number of techniques for eye tracking, also knownas gaze tracking. Some of these techniques determine a user's gazedirection from the orientation of the pupils of the user's eyes. Someknown eye gaze tracking techniques involve illuminating the eyes byemitting light from one or more light sources and detecting reflectionsof the emitted light off of the corneas with a sensor. Typically, thisis accomplished using invisible light sources in the infrared range andcapturing image data (e.g., images or video) of the illuminated eyeswith an infrared sensitive camera. Image processing algorithms are thenused to analyze the image data to determine eye gaze direction.

Generally, eye tracking image analysis takes advantage ofcharacteristics distinctive to how light is reflected off of the eyes todetermine eye gaze direction from the image. For example, the image maybe analyzed to identify eye location based on corneal reflections in theimage data, and the image may be further analyzed to determine gazedirection based on a relative location of the pupils in the image.

Two common gaze tracking techniques for determining eye gaze directionbased on pupil location are known as Bright Pupil tracking and DarkPupil tracking. Bright Pupil tracking involves illumination of the eyeswith a light source that is substantially in line with the optical axisof the camera, causing the emitted light to be reflected off of theretina and back to the camera through the pupil. The pupil presents inthe image as an identifiable bright spot at the location of the pupil,similar to the red eye effect which occurs in images during conventionalflash photography. In this method of gaze tracking, the brightreflection from pupil itself helps the system locate the pupil ifcontrast between pupil and iris is not enough.

Dark Pupil tracking involves illumination with a light source that issubstantially off line from the optical axis of the camera, causinglight directed through the pupil to be reflected away from the opticalaxis of the camera, resulting in an identifiable dark spot in the imageat the location of the pupil. In alternative Dark Pupil trackingsystems, an infrared light source and cameras directed at eyes can lookat corneal reflections. Such camera based systems track the location ofthe pupil and corneal reflections which provides parallax due todifferent depths of reflections gives additional accuracy.

FIG. 1A depicts an example of a dark pupil gaze tracking system 100 thatmay be used in the context of the present disclosure. The gaze trackingsystem tracks the orientation of a user's eye E relative to a displayscreen 101 on which visible images are presented. While a display screenis used in the example system of FIG. 1A, certain alternativeembodiments may utilize an image projection system capable of projectingimages directly into the eyes of a user. In these embodiments, theuser's eye E would be tracked relative to the images projected into theuser's eyes. In the example of FIG. 1A, the eye E gathers light from thescreen 101 through a variable iris I and a lens L projects an image onthe retina R. The opening in the iris is known as the pupil. Musclescontrol rotation of the eye E in response to nerve impulses from thebrain. Upper and lower eyelid muscles ULM, LLM respectively controlupper and lower eyelids UL,LL in response to other nerve impulses.

Light sensitive cells on the retina R generate electrical impulses thatare sent to the user's brain (not shown) via the optic nerve ON. Thevisual cortex of the brain interprets the impulses. Not all portions ofthe retina R are equally sensitive to light. Specifically,light-sensitive cells are concentrated in an area known as the fovea.

The illustrated image tracking system includes one or more infraredlight sources 102, e.g., light emitting diodes (LEDs) that directnon-visible light (e.g., infrared light) toward the eye E. Part of thenon-visible light reflects from the cornea C of the eye and partreflects from the iris. The reflected non-visible light is directedtoward a suitable sensor 104 (e.g., an infrared camera) by awavelength-selective mirror 106. The mirror transmits visible light fromthe screen 101 but reflects the non-visible light reflected from theeye.

The sensor 104 is preferably an image sensor, e.g., a digital camerathat can produce an image of the eye E which may be analyzed todetermine a gaze direction GD from the relative position of the pupil.This image may be produced with a local processor 120 or via thetransmission of the obtained gaze tracking data to a remote computingdevice 160. The local processor 120 may be configured according towell-known architectures, such as, e.g., single-core, dual-core,quad-core, multi-core, processor-coprocessor, cell processor, and thelike. The image tracking data may be transmitted between the sensor 104and the remote computing device 160 via a wired connection (not shown),or wirelessly between a wireless transceiver 125 included in the eyetracking device 110 and a second wireless transceiver 126 included inthe remote computing device 160. The wireless transceivers may beconfigured to implement a local area network (LAN) or personal areanetwork (PAN), via a suitable network protocol, e.g., Bluetooth, for aPAN.

The gaze tracking system 100 may also include an upper sensor 108 andlower sensor 109 that are configured to be placed, for example,respectively above and below the eye E. Sensors 108 and 109 may beindependent components, or may alternatively be part of a component 110worn on the user's head that may include, but is not limited to, anycombination of the sensor 104, local processor 120, or inertial sensor115 described below. In the example system shown in FIG. 1A, sensors 108and 109 are capable of collecting data regarding the electrical impulsesof the nervous system and/or the movement and/or vibration of themuscular system from those areas surrounding the eye E. This data mayinclude for example, electrophysiological and/or vibrational informationof the muscles and/or nerves surrounding the eye E as monitored by theupper sensor 108 and lower sensor 109. The electrophysiologicalinformation collected by sensors 108 and 109 may include, for example,electroencephalography (EEG), electromyography (EMG), or evokedpotential information collected as a result of nerve function in thearea(s) surrounding the eye E. Sensors 108 and 109 may also be capableof collecting, for example, mechanomyogram or surface electromyograminformation as a result of detecting the muscular vibrations or twitchesof the muscles surrounding the eye E. The data collected by sensors 108and 109 may be delivered with the image tracking data to the localprocessor 120 and/or the remote computing device 160 as described above.

The gaze tracking system 100 may also be capable of tracking a user'shead. Head tracking may be performed by an inertial sensor 115 capableproducing signals in response to the position, motion, orientation orchange in orientation of the user's head. This data may be sent to thelocal processor 120 and/or transmitted to the remote computing device160. The inertial sensor 115 may be an independent component, or mayalternatively be part of a component 110 worn on the user's head thatmay include, but is not limited to, any combination of the sensor 104,local processor 120, or sensors 108 and 109 described above. Inalternative embodiments, head tracking may be performed via the trackingof light sources on the component 110.

The remote computing device 160 may be configured to operate incoordination with the eye tracking device 110 and the display screen101, in order to perform eye gaze tracking and determine lightingconditions in accordance with aspects of the present disclosure. Thecomputing device 160 may include one or more processor units 170, whichmay be configured according to well-known architectures, such as, e.g.,single-core, dual-core, quad-core, multi-core, processor-coprocessor,cell processor, and the like. The computing device 160 may also includeone or more memory units 172 (e.g., random access memory (RAM), dynamicrandom access memory (DRAM), read-only memory (ROM), and the like).

The processor unit 170 may execute one or more programs, portions ofwhich may be stored in the memory 172, and the processor 170 may beoperatively coupled to the memory 172, e.g., by accessing the memory viaa data bus 178. The programs may be configured to perform eye gazetracking and determine lighting conditions for the system 100. By way ofexample, and not by way of limitation, the programs may include gazetracking programs 173, the execution of which may cause the system 100to track a user's gaze, e.g., as discussed above, error and stateparameter determination programs 174, and foveation rendering programs175, the execution of which render foveated images to be presented onthe display. The foveation rendering programs 175 may use error and/orstate parameters to determine potential adjustments that can be made toimages presented and to adjust the foveation of images to be presentedon the display, respectively, e.g., as discussed below with respect toFIG. 2.

By way of example, and not by way of limitation, the gaze trackingprograms 173 may include processor executable instructions which causethe system 100 to determine one or more gaze tracking parameters of thesystem 100 from eye tracking data gathered with the image sensor 104 andeye movement data gathered from the upper and lower sensors 108 and 109,respectively, while light is emitted from the lighting source 102. Thegaze tracking programs 173 may also include instructions which analyzeimages gathered with the image sensor 104 in order to detect a presenceof a change in lighting conditions, e.g., as described below withrespect to FIG. 2.

As seen in FIG. 1B, the image 181 showing a user's head H may beanalyzed to determine a gaze direction GD from the relative position ofthe pupil. For example, image analysis may determine a 2-dimensionaloffset of the pupil P from a center of the eye E in the image. Thelocation of the pupil relative to the center may be converted to a gazedirection relative to the screen 101, by a straightforward geometriccomputation of a three-dimensional vector based on the known size andshape of the eyeball. The determined gaze direction GD is capable ofshowing the rotation and acceleration of the eye E as it moves relativeto the screen 101.

As also seen in FIG. 1B, the image may also include reflections 187 and188 of the non-visible light from the cornea C and the lens L,respectively. Since the cornea and lens are at different depths, theparallax and refractive index between the reflections may be used toprovide additional accuracy in determining the gaze direction GD. Anexample of this type of eye tracking system is a dual Purkinje tracker,wherein the corneal reflection is the first Purkinje Image and the lensreflection is the 4th Purkinje Image. There may also be reflections 190from a user's eyeglasses 193, if these are worn a user.

Current HMD panels refresh at a constant rate of 90 or 120 Hertz (Hz)depending on the manufacturer. The high refresh rate increases powerconsumption of the panel and bandwidth requirements of the transmissionmedium to send frame updates. However, the image displayed on the paneldoes not always need to be refreshed during events that interrupt theuser's visual perception. For example, when a user blinks, visualinformation is shut off by the eyelids. When a user's eyes undergo asaccade, the brain effectively shuts off interpretation of visualinformation. Human eyes also exhibit rapid eye movements known assaccades. A phenomenon known as saccadic masking occurs during asaccade. Saccadic masking causes the brain to suppress visualinformation during eye movements. Power and computational resourcesdevoted to rendering frames during a vision interrupting event, such asa saccade or blink, are therefore wasted.

Many types of displays, such as HMD systems, are strobed systems thatutilize persistence of vision to keep the image stable. There is arelatively large variation in the duration of a saccade or blink. Forexample, a saccade typically lasts from 20 to 200 ms. This correspondsto between 2 and 25 frames at a frame rate of 120 frames per second(fps). Even if it takes 10 ms to detect the start of saccade and thesaccade only lasts 20 ms, the graphics system can save one frame, e.g.,not render to reduce computation or turn off the display to save poweror both. A blink typically lasts from about 100 ms to about 150 ms,which sufficient time for 12 to 18 frames at 120 fps.

As discussed with respect to FIG. 1A, camera-based eye tracking can beaugmented with other methods to update eye tracking during a blinkphase. Examples of augmentation include providing EEG information inaddition to the image information in order to detect nerve impulses thattrigger eye muscle activity. This information can also be used helpdetect the start and end of blinks and saccades, or to predict theduration of blinks and saccades. Eye tracking systems can determinewhether the vision system is in a saccade or not by high-pass filteringbased on rate of eye movement.

With knowledge that a blink or a saccade has begun and when it will end,a graphics rendering system could selectively disable transmission offrames until the time a user's saccade or blink will finish. Then thesystem could schedule work to finish the transmission in time to updatethe display. The end result is a savings in computation time,transmission bandwidth, and power consumption. Computational resourcesthat would otherwise be devoted to graphics may be used for otherthings, e.g., physics simulations and AI processing for renderingsubsequent frames can run during this time.

For further computational savings, gaze tracking may also be analyzed topredict the user's gaze point on the display at the end of the saccadeor blink and render the frame using foveated rendering. Furthermore, asthe user uses the system over a period of time, software running on thecomputing device 160 can analyze gaze tracking data to improve detectionand estimates of duration estimation and final gaze point for saccadesor blinks.

FIG. 2 shows an example method 200 wherein a system could adjust thecompression or transmission of graphics transmitted to a user in waysthat take into account saccades and/or blinks by a viewer. In thisexample, gaze tracking data 202 is obtained as discussed with respect toFIGS. 1A-1B. The eye tracking data may then be analyzed to detect asaccade and/or blink, as indicated at 204. If, at 206 no saccade orblink is detected normal transmission of image data to the display maytake place, as indicated at 210A followed by presentation of images, asindicated at 212. The normal transmission takes place with normaltransmission parameters and/or data compression parameters. If instead,a saccade and/or blink is detected at 206 the transmission of image datamay be disabled at 210B for a period that accounts for the nature of thesaccade/blink, which may potentially include determining the duration ofthe saccade blink at 208 through analysis of the gaze tracking data.Determining the saccade/blink duration at 208 may also includepredicting when the saccade/blink will end by utilizing historical gazetracking data of the user. When the system determines that thesaccade/blink is ending, normal compression/transmission of the imagedata resumes, and the resulting images may then be presented at 212.

In alternative implementations the normal transmission at 210A mayinvolve compression of the transmitted images. For example, the imagesmay be selectively compressed based on additional parameters determinedfrom the gaze tracking data. For example, and not by way of limitation,the quantization parameters may be determined for each foveal region ofthe image presented to the user, and this parameter may be used toselectively compress the foveal regions of the image before transmissionand subsequent presentation to the user.

Furthermore, the disabling transmission may involve reducingtransmission power, ignoring packet loss or both. Normal compression mayresume, or transmission power may be increased and/or packet lossrecovery may be reinstituted so that images are transmitted normallyfollowing the saccade or blink.

FIG. 3 depicts an example system for eye tracking 300 to furtherillustrate various aspects of the present disclosure. The example system300 may include a computing device 360 which is coupled to an eyetracking device 302 and a display device 304 in order to perform eyegaze tracking and/or calibration for eye tracking in accordance withaspects of the present disclosure. The display device 304 may be in theform of a cathode ray tube (CRT), flat panel screen, touch screen, orother device that displays text, numerals, graphical symbols, or othervisual objects. According to aspects of the present disclosure, thecomputing device 360 may be an embedded system, mobile phone, personalcomputer, tablet computer, portable game device, workstation, gameconsole, and the like. Moreover, the computing device 360, the eyetracking device 302, the display device 304, or any combination thereofmay form an integral unit or be implemented as separate components whichmay be in communication with each other.

The eye tracking device 302 may be coupled to the computing device 360,and may include a dynamic lighting source 310 similar to light sources110 of FIGS. 1A-1B. By way of example, and not by way of limitation, thelighting source 310 may be an invisible lighting source in the form ofone or more infrared LEDs, which may be configured to illuminate auser's eyes in order to gather eye tracking data with the sensor 312.The sensor 312 of the eye tracking device may be a detector which issensitive to light emitted from the light source 310. For example, thesensor 312 may be a camera sensitive to the light source such as aninfrared camera, and the camera 312 may be positioned relative to theeye tracking device and the lighting source so that it may captureimages of an area illuminated by the lighting source 310.

The computing device 360 may be configured to operate in coordinationwith the eye tracking device 302 and the display device 304, in order toperform eye gaze tracking and determine lighting conditions inaccordance with aspects of the present disclosure. The computing device360 may include one or more processor units 370, which may be configuredaccording to well-known architectures, such as, e.g., single-core,dual-core, quad-core, multi-core, processor-coprocessor, cell processor,and the like. The computing device 360 may also include one or morememory units 372 (e.g., random access memory (RAM), dynamic randomaccess memory (DRAM), read-only memory (ROM), and the like).

The processor unit 370 may execute one or more programs, portions ofwhich may be stored in the memory 372, and the processor 370 may beoperatively coupled to the memory 372, e.g., by accessing the memory viaa data bus 376. The programs may be configured to perform eye gazetracking and determine lighting conditions for the system 300. By way ofexample, and not by way of limitation, the programs may include gazetracking programs 373, execution of which may cause the system 300 totrack a user's gaze, e.g., as discussed above with respect to FIG. 1,saccade/blink detection programs 374, execution of which analyze gazetracking information to determine onset and/or duration of a blink orsaccade, e.g., as discussed above with respect to FIG. 2, and renderingadjustment programs 375, execution of which varies graphics processingand rendering of images to be presented on the display device 304 toaccount for saccades and blinks, e.g., as implemented by a method havingone or more features in common with the method of FIG. 2. By way ofexample, and not by way of limitation, the gaze tracking programs 373may include processor executable instructions which cause the system 300to determine one or more gaze tracking parameters of the system 300 fromeye tracking data gathered with the camera 312 while light is emittedfrom the dynamic lighting source 310. The gaze tracking programs 373 mayalso include instructions which analyze images gathered with the camera312, e.g., as described above with respect to FIG. 1B.

The computing device 360 may also include well-known support circuits378, such as input/output (I/O) circuits 379, power supplies (P/S) 380,a clock (CLK) 381, and cache 382, which may communicate with othercomponents of the system, e.g., via the bus 376. The I/O circuits mayinclude a wireless transceiver to facilitate communication withsimilarly configured transceivers on the eye tracking device 302 anddisplay device 379. The processor unit 370 and wireless transceiver maybe configured to implement a local area network (LAN) or personal areanetwork (PAN), via a suitable network protocol, e.g., Bluetooth, for aPAN. The computing device 360 may optionally include a mass storagedevice 384 such as a disk drive, CD-ROM drive, tape drive, flash memory,or the like, and the mass storage device 384 may store programs and/ordata. The computing device 360 may also include a user interface 388 tofacilitate interaction between the system 300 and a user. The userinterface 388 may include a keyboard, mouse, light pen, game controlpad, touch interface, or other device.

The system 300 may also include a controller (not pictured) whichinterfaces with the eye tracking device 302 in order to interact withprograms executed by the processor unit 370. The system 300 may alsoexecute one or more general computer applications (not pictured), suchas a video game, which may incorporate aspects of eye gaze tracking assensed by the tracking device 302 and processed by the tracking programs373, foveation adjustment programs 374, and foveation rendering programs375.

The computing device 360 may include a network interface 390, configuredto enable the use of Wi-Fi, an Ethernet port, or other communicationmethods. The network interface 390 may incorporate suitable hardware,software, firmware or some combination thereof to facilitatecommunication via a telecommunications network. The network interface390 may be configured to implement wired or wireless communication overlocal area networks and wide area networks such as the Internet. Thenetwork interface 390 may also include the aforementioned wirelesstransceiver that facilitates wireless communication with the eyetracking device 302 and display device 379. The computing device 360 maysend and receive data and/or requests for files via one or more datapackets 399 over a network.

Foveated Rendering

In some implementations, foveated rendering may augment computationalresource savings from leveraging knowledge of blinks or saccadicmasking. Foveated rendering can reduce computational complexity inrendering graphics, while still preserving essential details in regionsof interest in the image presented by the display. Foveated renderingreduces computation by performing high resolution rendering on regionsof interest (ROI) of the displayed image where the fovea is focused andlow resolution rendering outside this region. In addition to loweringthe computational cost, one can also use these ROIs to compress theframe for transmission. To utilize foveated rendering, an image displaydevice, such as a head-mounted display (HMD) would use eye gaze trackingtechnology to determine where the user is focusing on the screen.

Foveated rendering may be implemented by adjusting certain parameters ofthe rendering process based on screen location. Such adjustment may,e.g., vary the pixel resolution of the rendered image based on screenlocation. Alternatively, the density of vertices used to renderthree-dimensional objects may vary by screen location.

Using Foveated rendering images as input to compression, one can usevarying levels of compression for each rendered region. The output isone or more compression streams with varying levels of compression orquality. For the foveal ROI, the highest quality settings are usedgiving minimal or no compression. However, for regions outside thefovea, the eye is less sensitive and therefore higher compression isacceptable. The result is a reduction in the bandwidth required forframe transmission while preserving quality of important regions.

FIGS. 4A-4B illustrate an example of adjustment of vertex density toimplement foveated rendering in the context of a Virtual Reality (VR)environment. In conventional FOV displays, three-dimensional geometry isrendered using a planar projection to the view plane. However, renderinggeometry onto displays, especially high FOV view planes, can be veryinefficient and result in significant latency and performance issues.These issues can cause the displayed frame rate to drop below desiredlevels, creating a jarring, non-immersive experience for a user, andpotentially inducing motion sickness in a user immersed in a VRenvironment.

Additionally, regions of the display near the edge of the screen, orregions which the user is not viewing, or not likely to view, hold muchless meaningful information than regions near the center or to which auser's attention is currently directed. When rendering a sceneconventionally, these regions have the same number of vertices and thetime spent rendering equal sized regions on the screen is the same.Other parts of the screen, such as the rear-view mirrors in the drivingscene depicted in FIG. 4A, may be more important. These parts of theimage are referred to as regions of interest 480.

FIG. 4B illustrates an example of a VR environment in which the sceneinformation is rendered using foveated tessellation in accordance withaspects of the present disclosure. By utilizing foveated tessellation ofreal-time graphics rendering, detail may be added and subtracted from a3D mesh for regions of interest 480 and corresponding silhouette edgesbased on a variety of parameters, e.g., camera distance, user attention,user eye movement, or depth of field. Detail in the areas surroundingregions of interest 480 can be defined as transition regions 482, anddetail in these areas may be rendered such that the areas contain lessdetail than the areas of interest 480 but more detail than theperipheral regions 483. This may be accomplished by rendering thetransition regions 482 to establish, for example, a mathematicalrelationship between the pixel density distributions of the area ofinterest 480 and the peripheral region 483 (See FIGS. 6C-6D, below).Such foveated tessellation can reduce computational load and orrendering time for an image. Reductions in computational load and/orrendering time may alternatively by achieved in other parts of thegraphics processing pipeline by selectively reducing the pixelresolution outside of the regions of interest 480.

Experiments have shown, e.g., that by utilizing foveated tessellation,the rendering time of a 3D mesh or wireframe can be reduced by a factorof roughly 4× or more, as fewer vertex computations are required inrendering the image in the tessellation and certain parts of thegraphics pipeline subsequent to tessellation.

In some implementations, subsequent graphics processing may utilize arasterization stage that approximates a projection of the vertices ontoa curved viewport. In such implementations, the density of the projectedvertices may be determined for selected portions of the screen spacecorresponding to the region(s) of interest 480, such that a higherdensity of vertices is present in the region(s) of interest, while thedensity of the projected vertices is lower in remaining regions of thescreen space. This can be accomplished by reducing the density ofvertices for portions of the screen that are determined to be outsidethe region(s) of interest 480. In alternative embodiments, the densityof vertices may be increased in selected portions of the screen spacesuch that a higher density of vertices is present in a portion orportions of interest, and the density of vertices in the remainingportion or portions of the screen space is not increased. Accordingly,aspects of the present disclosure utilize a screen space transformationof the type described above to reduce a GPU's computational load byeffectively reducing the number of vertex computations for the area ofthe screen space that is to be rendered.

Foveated rendering may be limited by the capabilities of the gazetracking system. Performance of gaze tracking systems depend on amultitude of factors, including the placement of light sources (IR,visible, etc.) and cameras, whether user is wearing glasses or contacts,HMD optics, frame rate, exposure time, camera optics, tracking systemlatency, rate of eye movement, shape of eye (which changes during thecourse of the day or can change as a result of movement), eyeconditions, e.g., lazy eye, macular degeneration, gaze stability,fixation on moving objects, scene being displayed to user, and user headmotion.

In systems and devices that utilize eye tracking, errors in eye trackingand associated latencies in tracking, as well as the inability to trackeye state information, cause these systems to need a much greater radiusof high resolution on the display than is theoretically needed topreserve the high fidelity for the user. This issue is particularlyprevalent in virtual reality systems, wherein performance is dependenton screen resolution. In such systems, high levels of rendering arerequired in order to maintain an ideal resolution, however, much ofrendering performed is unnecessary since a user's eyes only focus on asmall part of the screen. Foveated rendering techniques allow for asystem to provide high resolution to the foveal region and lowerresolution to transitional and/or peripheral regions outside the fovealregion. However, even for systems utilizing foveated rendering, therendered foveal region is often larger than necessary as compared to thetheoretical foveal region, as the region is rendered to account for thevariability in human vision. An example of this variability involves thespeed and accuracy of a user's saccade to fixation.

Aspects of the present disclosure address these problems with anadaptive foveated rendering technique. Using error bounds andinformation regarding the state of the eye collected with the eyetracking data the system could adjust the fovea rendering radius tocompensate for state changes or errors in the tracking results. Thefovea rendering radius may be adjusted with respect to state changesoccurring in real time, or alternatively, may be adjusted inanticipation of a state change. Additionally, using knowledge oflatencies in the system one could scale the fovea region. The end resultwould allow for more savings in rendering complexity while maintainingthe highest possible resolution for the user.

In order to provide the most accurate scaling of the fovea region andmaximize the savings in rendering complexity while maintaining thehighest possible resolution for the user, aspects of the presentdisclosure may be configured to determine the size and shape of thefoveal region in advance based on a “worst case” scenario that accountsfor the variability in human vision, determine estimates of error, stateinformation, and latencies during gaze tracking, and dynamically resizethe foveal region to provide the best balance of resolution quality andrendering performance.

According to aspects of the present disclosure, real-time adjustment offoveated rendering of an image containing one or more regions ofinterest may be implemented by a graphics processing method 500illustrated in FIG. 5A. To understand the context of the graphicsprocessing method, certain conventional elements of computer graphicsprocessing are shown. Specifically, a computer graphics program maygenerate three-dimensional object vertex data 501 for one or moreobjects in three-dimensional virtual space. The object vertex data 501may represent the coordinates of points in the virtual space thatcorrespond to points on the surfaces of geometric shapes that make upone or more objects. An object may be made up of one or more geometricshapes. The object vertex data 501 may define a set of object verticesthat correspond to points on the surfaces of one or more shapes thatmake up an object. By way of example, and not by way of limitation, eachgeometric shape may be represented by a shape identifier (e.g., cone,sphere, cube, pyramid, etc.), coordinates in virtual space for arelevant location of the object, e.g., coordinates of a centroid of theshape, and relevant geometric parameters for determining coordinates ofa point on the surface of the shape. By way of example, for the case ofa sphere, the relevant location could be the location of the center ofthe sphere and the relevant geometric parameter could be the radius ofthe sphere.

As indicated at 502, the object vertex data 501 may be subject to aprocess that projects the object vertices onto a screen space in aconventional manner for 3D graphics processing. In some implementations,the projection may approximate a projection of the vertices onto acurved viewport. Polygons may then be generated from the projectedvertices, as indicated at 504. The generation of polygons from theprojected vertices may be done in a conventional manner. Specifically,edges may be defined between selected pairs of polygons and selectededges may be associated together as polygons. The resulting polygon data503 includes information identifying the vertices and edges that make upthe polygons. The polygon data 503 is used by the method 500, whichtessellates the polygons represented by the polygon data in accordancewith aspects of the present disclosure.

The method 500 includes determining foveation data 505 for one or moreregions of interest of the screen space, as indicated at 506 anddetermining vertex density information 507V and/or pixel resolution date507P, as indicated at 508. The polygon data 503, foveation data 505 andvertex density information 507V are used to tessellate the polygons inaccordance with aspects of the present disclosure, as indicated at 510to produce tessellated vertex data 509. The resulting tessellated vertexdata is then used in subsequent graphics processing, as indicated at512.

Determining the foveation data 505 may involve obtaining the gazetracking data as indicated at 506A, determining gaze tracking errorand/or state parameters at 506B, and adjusting regions of interest at506C. Gaze tracking data may be obtained, e.g., as discussed above withrespect to FIG. 1A-1B, and FIG. 2. The size and/or shape of ROI may beadjusted to compensate for errors in the gaze tracking by determiningthe error bounds of gaze tracking data. The ROI may also be adjusted tocompensate for the state of the user's eye in the gaze tracking data bydetermining the state information parameters of the gaze tracking data.Such adjustment of foveated rendering is described in detail in U.S.patent application Ser. No. 15/086,645 filed the same date as thepresent application, the entire contents of which are incorporated byreference herein.

Gaze tracking error parameters determined at 506B may include aconfidence interval regarding the current gaze position, which may bedetermined by examining the rotational velocity and acceleration of auser's eye for change from last position. Alternatively, the gazetracking error and/or state parameters may include a prediction offuture gaze position determined by examining the rotational velocity andacceleration of eye and extrapolating the possible future positions ofthe user's eye. In general terms, the fixed sampling rate or exposuretime of the gaze tracking system may lead to a greater error between thedetermined future position and the actual future position for a userwith larger values of rotational velocity and acceleration. Toaccommodate for the larger error the size of the foveal region mayincrease accordingly.

The gaze tracking error parameters may also include a measurement of theeye speed, e.g., the rotation rate. For a slow moving eye, the region ofinterest may be adjusted at 506C to be smaller, and peripheral and/ortransition regions may be adjusted so that they are larger. For a fastmoving eye, the size of the foveal region may increase, and theperipheral and/or transition regions may be made smaller.

The regions of interest may also be adjusted at 506C based on stateparameters established from the metrics of a user's blink. During ablink, a user's vision may not be focused on the presented images for upto 20-30 frames. However, upon exiting the blink, the user's gazedirection may not correspond to the last measured gaze direction asdetermined from the gaze tracking data. The metrics of a user's blink orblinks may be established from the gaze tracking data and regions ofinterest for subsequent images may be adjusted based on those metrics.For example, the metrics may include, but are not limited to, themeasured start and end times of the blink of a user, as well as thepredicted end times. The adjustment may involve, for example, decreasingthe size of the foveal region and increasing the size of the peripheraland/or transition regions during the blink, and increasing the size ofthe foveal region and decreasing the size of the peripheral and/ortransition regions as the blink is determined or predicted to be endingas a result of the blink cycle data.

Gaze tracking state parameters may also be related to saccades. A user'sgaze direction will have shifted to a different region of interest whenthe saccade is exited. The metrics of a user's saccade(s) may beestablished from the gaze tracking data 506A. These metrics may include,but are not limited to, the measured start and end times of the saccadesof a user as well as the predicted end times. The regions of interestfor subsequent images may accordingly be adjusted at 506C, e.g., basedon the predicted time that will elapse during the saccade. This mayinvolve, for example, decreasing the size of the foveal region whileincreasing the size of the peripheral and/or transition regions duringthe saccade, and increasing the size of the foveal region and decreasingthe size of the peripheral and/or transition regions as the saccade isdetermined to be ending as a result of the saccade cycle data.Alternatively, the foveal region may be eliminated completely when itdetermined that a saccade is either occurring or about to occur, and anew foveal region and peripheral/transition region boundaries may beestablished based on gaze tracking data 506A obtained during thesaccade. Gaze tracking state parameters may also account for atransition in gaze direction between areas of interest as a result of achange in depth of field between presented images that triggers asaccade.

Gaze tracking state parameters may be used to adapt for color blindness.For example, regions of interest may be present in an image presented toa user such that the regions would not be noticeable by a user who has aparticular color blindness. Gaze tracking data may be analyzed todetermine whether or not the user's gaze identified or responded thearea of interest, for example, as a result of the user's changed gazedirection. The region or regions of interest in subsequent imagespresented to a color blind user may be adjusted order to account for theuser's condition, by, for example, utilizing a different color scheme insubsequently presented areas of interest.

Gaze tracking data may also be analyzed to provide a measurement of thegaze stability of a user. Gaze stability may be determined, e.g., bymeasuring the microsaccadic radius of the user's eye; smaller fixationovershoot and undershoot equates to a more stable gaze in a user.Accordingly, the regions of interest for subsequent images may beadjusted at to be smaller for a user with greater gaze stability, orlarger for a user with less gaze stability.

Gaze tracking error or state parameters may also measure a user'sability to fixate on moving objects. These parameters may include themeasurement of the capability of a user's eye to undergo smooth pursuitand the maximum object pursuit speed of the eyeball. Typically, a userwith excellent smooth pursuit capabilities experiences less jitter inthe movement of the eyeball. The region of interest in subsequent imagesmay be adjusted correspondingly at to decrease the size of the regionwhere a user experiences less jitter, or increased where a userexperiences increased jitter. The region may also be adjusted at 506C inaccordance with a maximum pursuit speed of a user's eye, as a fastermeasured pursuit speed would require a larger region of interest ascompared to the region of interest necessary for a person with a slowerpursuit speed. Gaze tracking error parameters may also includedetermination of eye movement as a precursor to head movement. Offsetbetween head and eye orientation can affect certain error parameters asdiscussed above, e.g., in smooth pursuit or fixation. As a result, alarger offset between head an eye orientation may require the adjustmentof a region of interest for a subsequent image so to make the regionlarger, whereas a smaller offset would result in a smaller region ofinterest.

Once the adjustments at 506C have taken place, foveated images may begenerated and presented to the user. By way of example, and not by wayof limitation, in tessellating the polygons at 510, the foveation data505 and vertex density information 207V may define tessellationparameters that vary with respect to location in screen space and areused by a hardware or software tessellator to generate a triangle-basedtessellation of the polygons. Examples of such tessellation parametersinclude the so-called TessFactor, which controls the degree of finenessof the mesh generated by the Direct3D 11 programmable graphics pipeline,which is part of Windows 7 from Microsoft Corporation.

In general terms the foveation data 505 and vertex density information507V are used to modify a conventional tessellation process to accountfor the fact that not all regions of the screen space are equallyimportant to the one who views images of the screen space on a display.The foveal regions represent portions of the screen space that aredetermined by an application to be important to the viewer and aretherefore allocated a greater share of available graphics computationresources. The foveal region data 205 may include informationidentifying a location of a centroid of the foveal region in the screenspace, a size of the foveal region relative to the screen space, andshape of the foveal region. A foveal region may be determined at 506 byan application to be of interest to a viewer because (a) it is a regionthe viewer is likely look at, (b) it is a region the viewer is actuallylooking at, or (c) it is a region it is desired to attract the user tolook at.

With respect to (a), the foveal region may be determined to be likely tobe looked at in a context sensitive manner. In some implementations, theapplication may determine that certain portions of the screen space orcertain objects in a corresponding three-dimensional virtual space are“of interest” and such objects may be consistently drawn using a greaternumber of vertices than other objects in the virtual space. Fovealregions may be contextually defined to be of interest in a static ordynamic fashion. As a non-limiting example of static definition, afoveal region may be a fixed part of the screen space, e.g., a regionnear the center of the screen, if it is determined that this region isthe part of the screen space that a viewer is most likely to look at.For example, if the application is a driving simulator that displays animage of a vehicle dashboard and a windshield, the viewer is likely tobe looking at these portions of the image. In this example, the fovealregion may be statically defined in the sense that the region ofinterest is a fixed portion of the screen space. As a non-limitingexample of dynamic definition, in a video game a user's avatar, fellowgamer's avatars, enemy artificial intelligence (AI) characters, certainobjects of interest (e.g., the ball in a sports game) may be of interestto a the user. Such objects of interest may move relative to the screenspace and therefore the foveal region may be defined to move with theobject of interest.

With respect to (b) it is possible to track the viewers gaze todetermine which portion of a display the viewer is looking at. Trackingthe viewer's gaze may be implemented by tracking some combination of theuser's head pose and the orientation of the pupils of the user's eyes.Some examples of such gaze tracking are described e.g., in U.S. PatentApplication Publications Numbers 2015/0085250, 2015/0085251, and2015/0085097, the entire contents of all of which are incorporatedherein by reference. Further details of estimation of head pose can befound, e.g., in “Head Pose Estimation in Computer Vision: A Survey” byErik Murphy, in IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINEINTELLIGENCE, Vol. 31, No. 4, April 2009, pp 607-626, the contents ofwhich are incorporated herein by reference. Other examples of head poseestimation that can be used in conjunction with embodiments of thepresent invention are described in “Facial feature extraction and posedetermination”, by Athanasios Nikolaidis Pattern Recognition, Vol. 33(Jul. 7, 2000) pp. 1783-1791, the entire contents of which areincorporated herein by reference. Additional examples of head poseestimation that can be used in conjunction with embodiments of thepresent invention are described in “An Algorithm for Real-time StereoVision Implementation of Head Pose and Gaze Direction Measurement”, byYoshio Matsumoto and Alexander Zelinsky in FG '00 Proceedings of theFourth IEEE International Conference on Automatic Face and GestureRecognition, 2000, pp 499-505, the entire contents of which areincorporated herein by reference. Further examples of head poseestimation that can be used in conjunction with embodiments of thepresent invention are described in “3D Face Pose Estimation from aMonocular Camera” by Qiang Ji and Ruong Hu in Image and VisionComputing, Vol. 20, Issue 7, 20 Feb. 2002, pp 499-511, the entirecontents of which are incorporated herein by reference.

With respect to (c), it is a common cinematic device to change the depthof focus of a scene to focus on a portion of interest, e.g., aparticular actor who is speaking. This is done to draw the viewer'sattention to the portion of the image that is in focus. According toaspects of the present disclosure, a similar effect may be implementedwith computer graphics by moving the foveal region to a desired portionof the screen so that that portion has a greater density of vertices andis rendered in greater detail as a result.

In addition to locating a centroid, determining the foveal region dataat 506 may also involve determining the size and shape of the fovealregion relative to the screen space at run time. The shape of the fovealregion, e.g., circular, elliptical, arbitrary may be initialized inadvance, and this foveal region may be adjusted dynamically at run-time.In alternative embodiments, the shape of the foveal region is notpredetermined, but is established dynamically. In embodiments whereinthe shape of the foveal region is initialized in advance, the size ofthe foveal region may depend on a distance of the viewer from the screenand the size of the screen. Generally, the larger the screen and thecloser the viewer is to the screen the smaller the foveal regionrelative to the screen size. Conversely, the smaller the screen and thefurther the viewer is from the screen the larger the foveal regionrelative to the screen size.

In some implementations, as an alternative to adjusting the tessellationof the polygons, or in addition to it, the method 500 may involveadjusting the pixel resolution according to screen space location usingthe pixel resolution information 507P.

For fixed displays, such as television sets, tablet computer displays,smart phone displays, and computer monitors, screen size is fixed andmay be determined from metadata about the display. Such metadata may beexchanged when the display is connected to a processing system, such asa computer or gaming console. For projection type displays the size ofthe screen may be determined from additional information regarding thedistance from the projector to the wall or other surface on which imagesare projected.

In alternative implementations of the present disclosure, real-timeadjustment of foveated rendering of an image containing one or moreregions of interest may be implemented by a graphics processing method500 a illustrated in FIG. 5B. In FIG. 5B, the conventional elements ofcomputer graphics processing are not shown, but the final image to bedisplayed 511 may be generated with respect to the graphics processingmethod described in FIG. 5A.

The method 500 a of FIG. 5B uses selective image compression for theregions of interest and portions of the image outside the regions ofinterest. The selective image compression uses gaze position toselectively compress different regions around the fovea. The goal of theselective compression is to transmit fewer bits for the portions of theimage outside the regions of interest. For example, the imagecompression data can specify a higher quantization parameter forportions of an image that are outside the regions of interest than arespecified for the regions of interest. As a result, the portions of theimage outside the regions of interest undergo a higher degree ofcompression that reduces the number of bits that need to be transmittedbut at a cost of lower quality.

The method 500 a includes determining foveation data 505 for one or moreregions of interest of the screen space, as indicated at 506, anddetermining image compression (e.g., quantization parameter) data 507Qfor the regions of interest as indicated at 508 a. The foveation data505 may be determined as described with respect to FIG. 5A. The image511, foveation data 505 and image compression data 507Q are used tocompress the regions of interest of the image 511, as indicated at 513.The image compression data 507Q is configured so that fewer bits need tobe transmitted for regions of the image outside the regions of interestso that data transmission may be concentrated on the more importantparts of the image. By way of example, the regions of interest undergoless compression than portions of the image outside the regions ofinterest. As a result, more bits are used to encode and transmit theregions of interest and fewer bits are used to encode other portions ofthe image. In alternative implementations, error packet transmission maybe omitted for portions of the image outside the regions of interest butnot for the regions of interest. The resulting image data is thentransmitted via a bitstream, as shown at 515, to either a display orprojection system in accordance with certain embodiments of the presentdisclosure. The bitstream is then decompressed at 517, and the frame ispresented to the user at 519.

FIG. 6A illustrates an example of determining a foveal region by screenlocation in accordance with the above described aspects. In the exampleshown in FIG. 6A, the screen space is divided into subsections of equalsize in pixels. One or more central subsections of an example screenspace are desired to maintain or increase the density of tessellatedvertices, whereas subsections further from the center have progressivelylower densities of tessellated vertices. The foveation data 505 mayspecify which screen space subsections are part of the region ofinterest 480 and which are not, wherein the centroid of the region ofinterest is shown at 481 and the tracked gaze data is shown at 484. Insome implementations there may be two or more foveal regions of interestspecified by the foveation data 505. The areas of interest may besurrounded by transition areas 482, which may also be specified by thefoveation data 505 and which provide a buffer between the highlydetailed region of interest 480 and the peripheral region(s) 600. Thevertex density information 507V and/or pixel resolution information 507Pmay be adjusted in accordance with the foveation data so that thedensity of vertices and/or pixel resolution is highest in the subsectionor subsections of the screen containing a portion of interest 480 andprogressively lower in subsections further away from the foveal portion,such as the transition area(s) 482 and the peripheral areas 600described above and additionally shown in FIG. 4B. By way of example,and not by way of limitation, a tessellation factor for a given inputpolygon (e.g., patch) may be adjusted based on its importance (e.g., asdetermined from the location of the input polygon in the screen spaceand the foveation data 505). A tessellation unit, which may beimplemented in hardware or software, then generates output polygons(e.g., triangles) at 510 with a desired vertex density as determined bythe tessellation factor.

In conjunction with determining the foveal region, the density ofvertices and/or pixel resolution in the foveal region and outside thefoveal region may be determined, as indicated at 508. By way of example,and not by way of limitation, the vertex density information 507V and/orpixel resolution information 507P may include a maximum density for thefoveal region and a minimum density for regions outside the fovealregion may be determined. The terms “maximum density” and “minimumdensity” are used herein as terms of convenience. The maximum generallyrefers to a density distribution for the foveal region(s) having ahigher average vertex and/or pixel density than a corresponding densitydistribution for the remaining screen space regions. Likewise, theminimum density generally refers to a density distribution for theremaining region(s) that have a lower average vertex density than acorresponding density distribution for the foveal screen spaceregion(s). The transition area(s) 482 may have a pixel densitydistribution determined by, for example, a mathematical relationshipbetween the pixel density distributions of the area of interest 480 andthe peripheral region 483 (See FIGS. 6C-6D, below).

Vertex density and/or pixel resolution values as functions of locationin screen space (e.g., maximum and minimum values) may be fixed inadvance and stored in memory, in which case determining the valuesduring graphics processing is a trivial matter of retrieving the valuesfrom memory. The vertex density and/or pixel resolution values maydepend on a number of factors including (a) the available pixelresolution of the screen space, (b) the maximum available graphicsprocessing load capability, (c) the proximity of the viewer to thescreen, (d) the size of the screen in pixels, and (e) the nature of thegraphics being presented.

With regard to (a), the higher the available pixel resolution in screenspace, the higher maximum and minimum density values may be. With regardto (b), greater available graphics processing load capability means thatcomputational savings from adjustment of the vertex density may be lesscritical, leading to a higher maximum and minimum density values. Areduction on graphics processing load capability means thatcomputational savings from adjustment of the vertex density and/or pixelresolution are more critical, leading to, e.g., a lower value for theminimum density and possibly for the maximum density as well.

With regard to (c), as the viewer moves closer to the screen the needfor detail in the foveal region increases (leading to a greater valuefor the maximum vertex density or pixel resolution) and the need fordetail outside the foveal region decreases (allowing for a smaller valuefor the minimum vertex density or pixel resolution). With regard to (d),as the screen size decreases the foveal region becomes larger relativeto the screen. Fewer pixels available on the screen generally means thatthe minimum vertex density or pixel resolution value cannot be made toosmall.

In certain implementations, the transition of vertex densities or pixelresolutions (or “falloff”) in the transition region 482 between thefoveal portion of the region or regions of interest 480 and theremaining portions of the screen may be defined with a closed loop,based on the available computational resources and the complexity of thescene. In certain implementations, a foveation steering element(determining which portion or portions of the screen are the portions ofinterest), starting and ending mesh density of a rendered scene, andfalloff function may be defined statically in advance. In alternativeembodiments, these elements may be defined dynamically based on asoftware agent in the program region that analyzes frame data todetermine points or regions of interest. In alternative embodiments,these elements may be predefined by the game developer.

FIGS. 6B, 6C, and 6D illustrate examples of “falloff” in vertex densityor pixel resolution between the foveal portions and remaining portionsof the screen space. FIG. 6B illustrates an example of a step functiontransition between Max and Min density with respect to distance from afoveal screen space region. In this case, the vertex density or pixelresolution has the maximum value within the foveal region and theminimum value elsewhere. In alternative implementations, the vertexdensity or pixel resolution has the maximum value within the region ofinterest 480, the minimum value in the peripheral region 483 and anintermediate value in the transition region 482. By way of example, andnot by way of limitation, the pixel resolution may be 1080P in theregion of interest 480, 720P in the transition region 482, and 480P inthe peripheral region 483.

FIG. 6C illustrates an example of a linear function transition betweenMax and Min density with respect to distance from foveal screen spaceregion. In a transition region 482 the density depends linearly ondistance from the region of maximum or minimum density, depending on howthe coordinate system is defined.

FIG. 6D illustrates an example of a sigmoidal (“S”-shaped) functiontransition 482 between Max and Min density with respect to distance fromfoveal screen space region. In general, the integral of any smooth,positive, “bump-shaped” function will be sigmoidal. Examples of sigmoidfunctions include, but are not limited to, the logistic function, thegeneralized logistic function, sigmoid functions include the ordinaryarctangent, the hyperbolic tangent, the Gudermannian function, and theerror function

${{{erf}(x)} = {\frac{2}{\sqrt{\pi}}{\int_{0}^{x}{e^{t^{2}}{dt}}}}},$the complementary error function (1−erf(x)), and algebraic functionslike

${f(x)} = {\frac{x}{\sqrt{1 + x^{2}}}.}$

The logistic function has the form

${{f(x)} = \frac{L}{1 + e^{- {k{({x - x_{0}})}}}}},$where:x₀=the x-value of the sigmoid midpoint,L=the curve's maximum value, andk=the steepness of the curve.

In additional alternative embodiments, these elements may be dynamicallydefined by an external signal or signals, e.g., from a gaze trackingsystem. Gaze tracking signals may include, but are not limited to, acombination of head and pupil tracking. In such embodiments, a user'spupils may be tracked with a camera, as discussed herein. In embodimentswherein the external signal includes head tracking, the tracking of theuser's head may include, but is not limited to tracking the user's headwith an inertial sensor and/or tracking of light sources on a HMDdevice. Alternatively the external signal or signals may include, butare not limited to, laser pointer tracking, finger tracking, headtracking, tracking with a controller or peripheral device, trackinganother player character in a VR environment, or detecting andinterpreting conversation between users.

According to aspects of the present disclosure, certain implementationsmay utilize existing surface subdivision software, e.g., open sourcesoftware such as Open Subdiv, to compute a smooth limit surface from asmall number of vertices. In such embodiments, polygon tessellation at510 may tessellate the foveal portion or regions of interest 480 tofollow the smooth surface limit. The remaining portions may betessellated using a larger error tolerance.

Performing subsequent graphics operations at 512 may include somethingas simple as storing the tessellated vertex data in a memory ortransmitting the tessellated vertex data to another processing system.In addition, such subsequent processing may include well-known stages ofthe graphics processing pipeline. By way of example, and not by way oflimitation, primitive assembly is performed on the tessellated verticesto generate a one or more primitives in screen space. Scan conversionmay be performed on the one or more primitives to determine which pixelor pixels are part of corresponding primitives. A finished frame maythen be generated by performing pixel processing to assign pixel valuesto the pixel or pixels. In these stages, the pixel resolution may beadjusted for the regions of interest 480 and/or portions outside them toreduce computational load and or rendering time. In someimplementations, the finished frame can be stored in the memory ordisplayed on the display device. Additional details of a graphicspipeline are discussed below with respect to FIG. 7 and FIG. 8.

Graphics Processing System and Apparatus

Aspects of the present disclosure include graphics processing systemsthat are configured to implement graphics processing in which effectiveresolution varies by screen location by adjusting a density of verticesfor selected portions of the screen-space with respect to portions ofthe screen space determined to be portions of interest. By way ofexample, and not by way of limitation, FIG. 7 illustrates a blockdiagram of a computer system 700 that may be used to implement graphicsprocessing according to aspects of the present disclosure. According toaspects of the present disclosure, the system 700 may be an embeddedsystem, mobile phone, personal computer, tablet computer, portable gamedevice, workstation, game console, and the like.

The system 700 generally includes a central processor unit (CPU) 702, agraphics processor unit (GPU) 704, and a memory 708 that is accessibleto both the CPU and GPU. The system 700 may also include well-knownsupport functions 710, which may communicate with other components ofthe system, e.g., via a data bus 709. Such support functions mayinclude, but are not limited to, input/output (I/O) elements 711, powersupplies (P/S) 712, a clock (CLK) 713 and cache 714. In addition to thecache 714, the GPU 704 may include its own GPU cache 714 _(G), and theGPU may be configured so that programs running on the GPU 704 canread-through or write-though the GPU cache 714 _(G).

The system 700 may be configured to obtain and analyze gaze trackinginformation for optimization of foveated rendering as discussed above.In some implementations, the system 700 may include the eye trackingdevice 302 with lighting source 312 and camera 312, as discussed abovewith respect to FIG. 3. Alternatively, the system may be configured tointeroperate with these components, e.g., via the I/O elements 711.

The system 700 may include the display device 716 to present renderedgraphics 717 to a user. In alternative implementations, the displaydevice 716 is a separate component that works in conjunction with thesystem, 700. The display device 716 may be in the form of a flat paneldisplay, head mounted display (HMD), cathode ray tube (CRT) screen,projector, or other device that can display visible text, numerals,graphical symbols or images. In particularly useful implementations, thedisplay 716 is a large field of view (FOV) device having a screen with afield of view of 90 degrees or more (e.g., 114 degrees or more). Thedisplay device 716 displays rendered graphic images 717 (e.g., finishedframes 760) processed in accordance with various techniques describedherein.

The system 700 may optionally include a mass storage device 715 such asa disk drive, CD-ROM drive, flash memory, tape drive, or the like tostore programs and/or data. The system 700 may also optionally include auser interface unit 718 to facilitate interaction between the system 700and a user. The user interface 718 may include a keyboard, mouse,joystick, light pen, game controller, or other device that may be usedin conjunction with a graphical user interface (GUI). The system 700 mayalso include a network interface 720 to enable the device to communicatewith other devices 725 over a network 722. The network 722 may be, e.g.,a local area network (LAN), a wide area network such as the internet, apersonal area network, such as a Bluetooth network or other type ofnetwork. These components may be implemented in hardware, software, orfirmware, or some combination of two or more of these.

Some virtual reality (VR) applications, e.g., multiplayer online videogames or virtual worlds, include a VR Social screen feature that allowsspectators at remote devices 725 to view the scene that the user of thesystem 700 sees. If the foveated rendering is specific to what the useris looking at, images with foveated rendering could be very confusing tospectators. To get around this, the system 700 may create a standard 2Dcompliant image without the foveated rendering. This image can be sharedwith spectators, e.g., via the social screen feature.

The CPU 702 and GPU 704 may each include one or more processor cores,e.g., a single core, two cores, four cores, eight cores, or more. Thememory 708 may be in the form of an integrated circuit that providesaddressable memory, e.g., RAM, DRAM, and the like. The memory 708 mayinclude a dedicated graphics memory 728 that may store graphicsresources and temporarily store graphics buffers 705 of data for agraphics rendering pipeline. The graphics buffers 705 may include, e.g.,vertex buffers VB for storing vertex parameter values, index buffers IBfor holding vertex indices, depth buffers (e.g., Z-buffers) DB forstoring depth values of graphics content, stencil buffers SB, framebuffers FB for storing completed frames to be sent to a display, andother buffers. In the example shown in FIG. 7, the graphics memory 728is shown as part of the main memory. In alternative implementations, thegraphics memory 728 could be a separate hardware component, possiblyintegrated into the GPU 704. The memory 708 (possibly the graphicsmemory 728) may also temporarily store the polygon data 503, foveationdata 505, vertex density data 507V, pixel resolution data 507P andtessellated vertex data 509.

By way of example, and not by way of limitation, the CPU 702 and GPU 704may access the memory 708 via the bus or busses 709. In some cases, itmay be useful for the system 700 to include two or more different buses.The memory 708 may contain data that can be accessed by the CPU 702 andGPU 704. The GPU 704 may include a plurality of compute units configuredto perform graphics processing tasks in parallel. Each compute unit mayinclude its own dedicated local memory store, such as a local datashare. Alternatively, the compute units may each access the memory 708or a dedicated graphics memory 728.

The CPU may be configured to execute CPU code 703 _(C), which mayinclude an application that utilizes graphics, a compiler and a graphicsAPI. The CPU code 703 _(C) may also implement gaze tracking andfoveation region adjustment, e.g., as discussed above with respect tothe method 200 of FIG. 2. The graphics API can be configured to issuedraw commands to programs implemented by the GPU. The CPU code 703 _(C)may also implement physics simulations and other functions. The GPU 704may be configured to operate as discussed above. In particular, the GPUmay execute GPU code 703 _(G), may include instructions configured toimplement the method 500 of FIG. 7, described above. In addition, theGPU code 703 _(G) may also implement well-known shaders, such as computeshaders CS, vertex shaders VS, and pixel shaders PS. To facilitatepassing of data between the compute shaders CS and the vertex shaders VSthe system may include one or more buffers 705, which may include aframe buffer FB. The GPU code 703 _(G) may also optionally implementother types of shaders (not shown), such as pixel shaders or geometryshaders. Each compute unit may include its own dedicated local memorystore, such as a local data share. The GPU 704 may include one or moretexture units 706 configured to perform certain operations for applyingtextures to primitives as part of a graphics pipeline.

According to certain aspects of the present disclosure, the CPU code 703_(c) and GPU code 703 _(g) and other elements of the system 700 areconfigured to implement a graphics pipeline in which the GPU 704 mayreceive the polygon data 503. The polygon data 503 can be generated fromcalculations, e.g., physics simulations of objects in athree-dimensional virtual space, implemented by execution of the CPUcode 703 _(C) by the CPU 702. The GPU 704 performs a projection of thepolygon vertices onto a screen space of the display device 716 andtessellation of the resulting projected polygons. The GPU 704 may adjustthe density of the vertices for the tessellation of the polygons inaccordance with the foveation data 505 and vertex density data 507V suchthat the density of vertices is higher in selected foveal portions ofthe screen space and lower in remaining portions. Alternatively, the GPU704 may adjust the pixel resolution for portions of images to bepresented on the display in accordance with the foveation data 505 andpixel resolution data 507P such that the pixel resolution is higher inselected regions of interest in the screen space and lower in remainingportions. In some implementations, the GPU 704 may adjust the both thevertex density and pixel resolution in accordance with the foveationdata 505, the vertex density data 507V and the pixel resolution data507P such that the vertex density and pixel resolution are higher inselected regions of interest in the screen space and lower in remainingportions.

The GPU 704 may then perform primitive assembly on the vertices togenerate a one or more primitives in screen space from the projection ofthe vertices onto the screen space. Scan conversion may then beperformed on the one or more primitives to determine which pixel ofscreen space are part of corresponding primitives. The GPU 704 may thengenerate a finished frame 760 by performing pixel processing to assignpixel values to the pixel or pixels that are part of the correspondingprimitives. The finished frame can be stored in the memory 708 orgraphics memory 728 (e.g., in the frame buffer FB) or displayed on thedisplay device 716.

The projection of the polygon vertices onto the screen space and otherrelated portions of the graphics pipeline can be performed in software,e.g., by a front end implemented as a compute shader CS. Alternatively,the projection of the vertices onto the screen space and other relatedportions of the graphics pipeline can be implemented by speciallydesigned hardware components HW that are configured to implement thesefunctions.

Aspects of the present disclosure also include implementations in whichthe foveation data 505 and/or vertex density data 507V and/or pixelresolution data 507P are adjusted dynamically. For example, the fovealregion may be defined in conjunction with gaze tracking. In suchimplementations, the system 700 includes hardware for tracking a user'sgaze, i.e., where a user's eye is pointing, and relating thisinformation to a corresponding screen location that the user is lookingat, e.g., as discussed above and further explained below. One example ofsuch hardware could include a digital camera in a known location withrespect to the screen of the display device 716 and pointed in thegeneral direction of a user. The digital camera could be part of theuser interface 718 or a separate component. The CPU code 703 _(C) couldinclude image analysis software that analyzes images from the camera todetermine (a) if the user is in the image; (b) if the user is facing thecamera; (c) if the user is facing the screen; (d) if the user's eyes arevisible; (e) the orientation of the pupils of the user's eyes relativeto the user's head; and (f) the orientation of the user's head relativeto the camera. From the known position and orientation of the camerawith respect to the screen, the orientation of the pupils of the user'seyes relative to the user's head and the orientation of the user's headrelative to the camera the image analysis software could determinewhether the user is looking at the screen and, if so, screen spacecoordinates for the portion of the screen the user is looking at. TheCPU code 703 _(c) could then pass these screen coordinates to the GPUcode 703 _(G), which could determine the subsection or subsectionscorresponding to one or more regions of interest. The GPU code 703 _(G)could then modify the adjustment of vertices and or pixel resolutionaccordingly so that the resolution is highest in the subsection orsubsections containing the foveal portion and progressively lower insubsections further away from the foveal portion, as shown in FIG. 6B.

By way of example, and not by way of limitation, specially designedhardware HW, the texture unit(s) 706, certain types of shaders, andother parts of the graphics pipeline described below may be implementedby special purpose hardware, such as an application-specific integratedcircuit (ASIC), Field Programmable Gate Array (FPGA), or a system onchip (SoC or SOC).

As used herein and as is generally understood by those skilled in theart, an application-specific integrated circuit (ASIC) is an integratedcircuit customized for a particular use, rather than intended forgeneral-purpose use.

As used herein and as is generally understood by those skilled in theart, a Field Programmable Gate Array (FPGA) is an integrated circuitdesigned to be configured by a customer or a designer aftermanufacturing-hence “field-programmable”. The FPGA configuration isgenerally specified using a hardware description language (HDL), similarto that used for an ASIC.

As used herein and as is generally understood by those skilled in theart, a system on a chip or system on chip (SoC or SOC) is an integratedcircuit (IC) that integrates all components of a computer or otherelectronic system into a single chip. It may contain digital, analog,mixed-signal, and often radio-frequency functions-all on a single chipsubstrate. A typical application is in the area of embedded systems.

A typical SoC includes the following hardware components:

-   -   One or more processor cores (e.g., microcontroller,        microprocessor or digital signal processor (DSP) cores.    -   Memory blocks, e.g., read only memory (ROM), random access        memory (RAM), electrically erasable programmable read-only        memory (EEPROM) and flash memory.    -   Timing sources, such as oscillators or phase-locked loops.    -   Peripherals, such as counter-timers, real-time timers, or        power-on reset generators.    -   External interfaces, e.g., industry standards such as universal        serial bus (USB), FireWire, Ethernet, universal asynchronous        receiver/transmitter (USART), serial peripheral interface (SPI)        bus.    -   Analog interfaces including analog to digital converters (ADCs)        and digital to analog converters (DACs).    -   Voltage regulators and power management circuits.

These components are connected by either a proprietary orindustry-standard bus. Direct Memory Access (DMA) controllers route datadirectly between external interfaces and memory, bypassing the processorcore and thereby increasing the data throughput of the SoC.

A typical SoC includes both the hardware components described above, andexecutable instructions (e.g., software or firmware) that controls theprocessor core(s), peripherals and interfaces.

In some implementations, some or all of the functions of parts of thegraphics pipeline may alternatively be implemented by appropriatelyconfigured software instructions executed by a software programmablegeneral purpose computer processor, e.g., as compute shaders CS executedby the GPU 704. Such instructions may be embodied in a computer-readablemedium, e.g., memory 708, graphics memory 728, or storage device 715.

Graphics Pipeline

According to aspects of the present disclosure, the system 700 may beconfigured to implement the method 200 of FIG. 2 and/or the method 500of FIG. 5 in conjunction with portions of a graphics rendering pipeline.FIG. 8 illustrates an example of a graphics rendering pipeline 730 inaccordance with aspects of the present disclosure.

The rendering pipeline 730 may be configured to render graphics asimages that depict a scene having a two-dimensional or preferablythree-dimensional geometry in virtual space (sometime referred to hereinas “world space”). The early stages of the pipeline may includeoperations performed in virtual space before the scene is rasterized andconverted to screen space as a set of discrete picture elements suitablefor output on the display device 716. Throughout the pipeline, variousresources contained in the graphics memory 728 may be utilized at thepipeline stages and inputs and outputs to the stages may be temporarilystored in buffers contained in the graphics memory before the finalvalues of the images are determined.

The rendering pipeline may operate on input data, such as the polygondata 503 that represents one or more virtual objects defined by a set ofvertices that are set up in virtual space and have geometry that isdefined with respect to coordinates in the scene. The early stages ofthe pipeline may include what is broadly categorized as a vertexprocessing stage 734 in FIG. 8, and this may include variouscomputations to process the vertices of the objects in virtual space.The vertex processing stage may include vertex shading computations 736,which may manipulate various parameter values of the vertices in thescene, such as position values (e.g., X-Y coordinate and Z-depthvalues), color values, lighting values, texture coordinates, and thelike. Preferably, the vertex shading computations 736 are performed byone or more programmable vertex shaders VS of the GPU 704. The vertexprocessing stage includes additional vertex processing computations,such as geometry shader computations 737 which project the polygonvertices onto the screen space and tessellation shader computations 738,which generate new vertices and new geometries from the projectedpolygon vertices. Tessellation computations subdivide scene geometriesand geometry shading computations to generate new scene geometriesbeyond those initially set up in the application implemented by the CPUcode 703 _(C). It is at this stage that foveated tessellation may occur,and the density of vertices in the 3D mesh or wireframe may be adjustedby tessellation shader computations 738 as defined in accordance withthe foveation data 505 and vertex density data 507V. Once the vertexprocessing stage 734 is complete, the scene is defined by a set oftessellated vertices represented by the tessellated vertex data 509. Inaddition each tessellated vertex has a set of vertex parameter values739. The vertex parameter values 739 may include texture coordinates,tangents, lighting values, colors, positions, and the like.

The pipeline 730 may then proceed to rasterization processing stages 740associated with converting the scene geometry into screen space and aset of discrete picture elements, i.e., pixels. The virtual spacegeometry (which can be three-dimensional) is transformed to screen spacegeometry (which is typically two-dimensional) through operations thatmay essentially compute the projection of the objects and vertices fromvirtual space to the viewing window (or “viewport) of the scene. Subsetsof the vertices are grouped to define sets of primitives in screenspace. In some implementations, e.g., for head-mounted displays, therasterization stage 740 may approximate a projection of the verticesonto a curved viewport, e.g., as described in U.S. patent applicationSer. No. 14/246,066, filed Apr. 5, 2014, which is incorporated herein byreference.

The graphics pipeline 730 may pass the foveation data 505 and pixelresolution data 507P to the rasterization processing stages 740 in orderto implement foveated rendering in these stages. This may be done as analternative to or in conjunction with foveated tessellation adjustmentof vertices in the 3D mesh or wireframe by the tessellation shadercomputations 738 defined in accordance with the foveation data 505 andvertex density data 507V.

In accordance the aspects of the present disclosure, the rasterizationstages 740 may include a front end 741. The front end can be implementedas part of the rasterization stage or as an intermediate stage betweenvertex processing 734 and the rasterization stage 740. In the exampledepicted in FIG. 8, the front end 741 is shown as part of therasterization stage 740. However, aspects of the present disclosure arenot limited to such implementations. In certain implementations, thefront end 741 is implemented wholly or partially as a compute shader CSrunning on the GPU 704. However, aspects of the present disclosure arenot limited to such implementations.

Operation of the front end 741 and related aspects of the presentdisclosure can be understood by referring simultaneously to FIG. 7 andFIG. 8.

The rasterization processing stage 740 depicted in FIG. 8 includesprimitive assembly operations 742, which may set up the primitivesdefined by each set of vertices in the scene. Each vertex may be definedby an index, and each primitive may be defined with respect to thesevertex indices, which may be stored in index buffers IB in the graphicsmemory 728. The primitives preferably include triangles defined by threevertices each, but may also include point primitives line primitives,and other polygonal shapes. During the primitive assembly stage 742,certain primitives may optionally be culled. For example, thoseprimitives whose indices indicate a certain winding order may beconsidered to be back-facing and may be culled from the scene.

By way of example, and not by way of limitation, where the primitivesare in the form of triangles defined by vertices in three dimensionalvirtual space, the primitive assembly determines where on the screen ofthe display 716 each triangle is located. Clipping and screen spacetransformation operations are typically performed by the primitiveassembly unit 742. The optional scan conversion operations 744 samplethe primitives at each pixel and generate fragments (sometimes referredto as pixels) from the primitives for further processing when thesamples are covered by the primitive. The scan conversion operationsinclude operations that take a primitive that has been converted toscreen space coordinates and determines which pixels are part of thatprimitive. In some implementations, multiple samples are taken withinthe primitives during the scan conversion operations 744, which may beused for anti-aliasing purposes. In certain implementations, differentpixels may be sampled differently. For example, some edge pixels maycontain a lower sampling density than center pixels to optimize certainaspects of the rendering for certain types of display device 716, suchas head mounted displays (HMDs). The fragments (or “pixels”) generatedfrom the primitives during scan conversion 744 may have parameter valuesthat may be interpolated to the locations of the pixels from the vertexparameter values 739 of the vertices of the primitive that created them.The rasterization stage 740 may include parameter interpolationoperations 746 stage to compute these interpolated fragment parametervalues 749, which may be used as inputs for further processing at thelater stages of the pipeline.

According to aspects of the present disclosure, between primitiveassembly 742 and scan conversion 744 certain operations may take placethat account for the fact that different subsections of the screen havedifferent pixel resolutions for foveated rendering. In particularimplementations, once the screen location for the vertices of aprimitive are known, a coarse rasterization 743 can be done to find allthe predefined screen subsections (sometimes referred to herein ascoarse rasterization tiles or supertiles) that the primitive overlaps.For each subsection that the primitive overlaps, the vertex locationsfor the primitive are adjusted to account for the pixel resolutions ofthe subsection. Scan conversion 744 and subsequent processing stagesgenerate the final pixel values by performing pixel processing only onthe specified number of active pixels for the relevant subsection orsubsections.

In certain implementations, the GPU 704 may be configured to implementcoarse division of primitives between subsections in software, and theprojection of the vertices, primitive assembly, and scan conversion inhardware. In some such implementations, the GPU 704 is configured toassociate subsection indices to primitive vertices in software, witheach subsection index selecting a screen space projection and viewportfrom a palette implemented in hardware. In other such implementations,the GPU 704 is configured to associate subsection indices to primitivevertex indices in software, with each subsection index selecting ascreen space projection and viewport from a palette implemented inhardware.

The graphics pipeline 730 further includes pixel processing operations,indicated generally at 750 in FIG. 8, to further manipulate theinterpolated parameter values 749 and perform further operationsdetermining how the fragments contribute to the final pixel values fordisplay 716. According to aspects of the present disclosure, these taskscan be performed in a conventional fashion. The pixel processing tasksinclude pixel shading computations 752 that further manipulate theinterpolated parameter values 749 of the fragments. The pixel shadingcomputations 752 may be performed by a programmable pixel shader orpurpose built hardware in the GPU 704. Pixel shader invocations 748 maybe initiated based on the sampling of the primitives during therasterization processing stages 740. The pixel shading computations 752may output values to one or more buffers 705 in graphics memory 728,sometimes referred to as render targets RT, or if multiple, as multiplerender targets (MRTs). MRTs allow pixel shaders to optionally output tomore than one render target, each with the same screen dimensions butpotentially with a different pixel format.

In some implementations, the adjusted foveation data may include dataused to determine pixel shading quality in the one or more regions ofinterest. This allows for dynamic foveated shader complexity. Forexample, slightly better parallax occlusion mapping shading in theregions of interest and simpler technique outside the regions ofinterest.

The pixel processing operations 750 typically include texture mappingoperations 754, which may be performed to some extent by one or moreshaders (e.g., pixel shaders PS compute shaders CS, vertex shaders VS orother types of shaders) and to some extent by the texture units 706. Thepixel shader computations 752 include calculating texture coordinates UVfrom screen space coordinates XY, and sending the texture coordinates tothe Texture Operations 754, and receiving texture data TX. The texturecoordinates UV could be calculated from the screen space coordinates XYin an arbitrary fashion, but typically are calculated from interpolatedinput values or sometimes from the results of previous textureoperations. Gradients Gr are often directly calculated from quads oftexture coordinates by the texture units 706 (Texture Operationshardware units), but can optionally be calculated explicitly by thepixel shader computations 752 and passed to the texture operations 754rather than relying on the texture units 706 to perform the defaultcalculation.

The texture operations 756 generally include the following stages, whichcan be performed by some combination of a pixel shader PS and a textureunit 406. First, one or more texture coordinates UV per pixel locationXY are generated and used to provide a coordinate set for each texturemapping operation. Gradient values Gr are calculated from the texturecoordinates UV and used to determine a level of detail (LOD) for atexture to apply to the primitive.

The pixel processing 750 generally culminates in render outputoperations 756, which may include what are commonly known as rasteroperations (ROP). Rasterization Operations (ROP) is simply run multipletimes per pixel, once for each render target among the multiple rendertargets (MRTs). During the output operations 756, the final pixel values759 may be determined in a frame buffer, which may optionally includemerging fragments, applying stencils, depth tests, and certain persample processing tasks. The final pixel values 759 include thecollected output to all active render targets (MRTs). The GPU 704 usesthe final pixel values 759 to make up a finished frame 760, which mayoptionally be displayed on the pixels of the display device 716 inreal-time.

According to aspects of the present disclosure, the graphics pipeline730 may implement modified rasterization processing 740 and/or modifiedpixel processing 750 in which the screen resolution and computationalload for processing of pixels varies according to screen space locationusing the foveation data 505 and pixel resolution data 507P. Examples ofsuch modified graphics processing are described e.g., in co-pending U.S.patent application Ser. Nos. 14/246,066 and 14/246,062, both of whichwere filed Apr. 5, 2014, the entire contents of both of which are hereinincorporated by reference. By modifying the graphics pipeline toconcentrate computational resources on foveal regions of the screenspace the overall computational load throughout the graphics pipelinemay be reduced. By way of example, in some implementations, the CPU code703 _(c), GPU code 703 _(G), and texture unit 706 may be furtherconfigured to implement modifications to texture mapping operations inconjunction with screen location dependent variable resolution. Inparticular, a pixel shader PS and texture unit 706 can be configured togenerate one or more texture coordinates UV per pixel location XY toprovide a coordinate set for one or more texture mapping operations,calculate gradient values Gr from the texture coordinates UV and use thegradient values to determine a level of detail (LOD) for a texture toapply to the primitive. These gradient values can be adjusted to accountfor the variable resolution as well as deviation from orthonormality inthe sample locations.

Although examples are described herein with respect to head mounteddisplay (HMD) applications, aspects of the present disclosure are notlimited to such implementations. HMD implementations represent arelatively straightforward implementation because the relative locationsof the user's eyes and the display screen remain more or less fixed. Inprinciple, however, the disclosed system and method may be adapted toany image display system that can work with gaze tracking. The gazetracking system may be modified to track the location and orientation ofthe user's head and eye(s) relative to the display screen inimplementations where these are not fixed.

FIGS. 9A-9H illustrate examples of the use of facial orientation and eyegaze direction in conjunction with aspects of the present disclosure. Asseen in FIG. 9A a face 920 of a user may appear in an image 922 _(A)obtained with a camera trained on the user. Such cameras are commonfeatures of devices such as laptop computers, smart phones, and tabletcomputers. Image analysis software may identify reference points on theface 920. The software may characterize certain of these referencepoints, e.g., located at the corners of the mouth 924 _(M), the bridgeof the nose 924 _(N), the part in the hair 924 _(H), and at the tops ofthe eyebrows 924 _(E), as being substantially fixed relative to the face920. The software may also identify the pupils 926 and corners 928 ofthe user's eyes as reference points and determine the location of thepupils relative to the corners of the eyes. In some implementations, thecenters of the user's eyes can be estimated from the locations of thepupils 926 and corners 928 of eyes. Then, the centers of eyes can beestimated and the locations of pupils can be compared with the estimatedlocations of the centers. In some implementations, face symmetryproperties can be used.

The software can determine the user's facial characteristics, e.g., headtilt angle and eye gaze angle from analysis of the relative locations ofthe reference points and pupils 926. For example, the software mayinitialize the reference points 924 _(E), 924 _(H), 924 _(M), 924 _(N),928 by having the user look straight at the camera and register thelocations of the reference points and pupils 926 as initial values. Thesoftware can then initialize the head tilt and eye gaze angles to zerofor these initial values. Subsequently, whenever the user looks straightahead at the camera, as in FIG. 9A and the corresponding top view shownin FIG. 9B, the reference points 924 _(E), 924 _(H), 924 _(M), 924 _(N),928 and pupils 926 should be at or near their initial values.

By way of example and not by way of limitation, the pose of a user'shead may be estimated using five reference points, the outside corners928 of each of the eyes, the outside corners 924 _(M) of the mouth, andthe tip of the nose (not shown). A facial symmetry axis may be found byconnecting a line between a midpoint of the eyes (e.g., halfway betweenthe eyes' outside corners 928) and a midpoint of the mouth (e.g.,halfway between the mouth's outside corners 924 _(M)). A facialdirection can be determined under weak-perspective geometry from a 3Dangle of the nose. Alternatively, the same five points can be used todetermine the head pose from the normal to the plane, which can be foundfrom planar skew-symmetry and a coarse estimate of the nose position.Further details of estimation of head pose can be found, e.g., in “HeadPose Estimation in Computer Vision: A Survey” by Erik Murphy, in IEEETRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, Vol. 31, No.4, April 2009, pp 607-626, the contents of which are incorporated hereinby reference. Other examples of head pose estimation that can be used inconjunction with embodiments of the present invention are described in“Facial feature extraction and pose determination”, by AthanasiosNikolaidis Pattern Recognition, Vol. 33 (Jul. 7, 2000) pp. 1783-1791,the entire contents of which are incorporated herein by reference.Additional examples of head pose estimation that can be used inconjunction with embodiments of the present invention are described in“An Algorithm for Real-time Stereo Vision Implementation of Head Poseand Gaze Direction Measurement”, by Yoshio Matsumoto and AlexanderZelinsky in FG '00 Proceedings of the Fourth IEEE InternationalConference on Automatic Face and Gesture Recognition, 2000, pp 499-505,the entire contents of which are incorporated herein by reference.Further examples of head pose estimation that can be used in conjunctionwith embodiments of the present invention are described in “3D Face PoseEstimation from a Monocular Camera” by Qiang Ji and Ruong Hu in Imageand Vision Computing, Vol. 20, Issue 7, 20 Feb. 2002, pp 499-511, theentire contents of which are incorporated herein by reference.

When the user tilts his head, the relative distances between thereference points in the image may change depending upon the tilt angle.For example, if the user pivots his head to the right or left, about avertical axis Z the horizontal distance x₁ between the corners 928 ofthe eyes may decrease, as shown in the image 922 _(C) depicted in FIG.9C. Other reference points may also work, or be easier to detect,depending on the particular head pose estimation algorithm being used.The amount change in the distance can be correlated to an angle of pivotθ_(H) as shown in the corresponding top view in FIG. 1E. It is notedthat if the pivot is purely about the Z axis the vertical distance Y₁between, say, the reference point at the bridge of the nose 924 _(N) andthe reference points at the corners of the mouth 924 _(M), would not beexpected to change significantly. However, it would be reasonablyexpected for this distance y₁ to change if the user were to tilt hishead upwards or downwards. It is further noted that the software maytake the head pivot angle θ_(H) into account when determining thelocations of the pupils 926 relative to the corners 928 of the eyes forgaze direction estimation. Alternatively the software may take thelocations of the pupils 926 relative to the corners 928 of the eyes intoaccount when determining head pivot angle θ_(H). Such an implementationmight be advantageous if gaze prediction is easier, e.g., with aninfrared light source on a hand-held device, the pupils could be locatedrelatively easily. In the example, shown in FIG. 9C and FIG. 9D, theuser's eye gaze angle θ_(E) is more or less aligned with the user's headtilt angle. However, because of the pivoting of the user's head and thethree-dimensional nature of the shape of the eyeballs, the positions ofthe pupils 926 will appear slightly shifted in the image 922 _(D)compared to their positions in the initial image 922 _(A).

In some situations, the user may be facing the camera, but the user'seye gaze is directed elsewhere, e.g., as shown in FIG. 9E and thecorresponding top view in FIG. 9F. In this example, the user's head istilt angle θ_(H) is zero but the eye gaze angle θ_(E) is not. Instead,the user's eyeballs are rotated counterclockwise, as seen in FIG. 9F.Consequently, the reference points 924 _(E), 924 _(H), 924 _(M), 124_(N), 928 are arranged as in FIG. 9A, but the pupils 126 are shifted tothe left in the image 922 _(E).

It is noted that the user's head may pivot in one direction and theuser's eyeballs may pivot in another direction. For example, asillustrated in FIG. 9H and FIG. 9I, the user 101 may pivot his headclockwise and rotate his eyeballs counterclockwise. Consequently, thereference points 924 _(E), 924 _(H), 924 _(M), 924 _(N), 928 are shiftedas in FIG. 9D, but the pupils 926 are shifted to the right in the image922 _(G) shown in FIG. 9G. The gaze tracking system 100, as described inFIGS. 1A-1B, may take this configuration or any of the configurationsdescribed above into account in determining the gaze direction GD of theuser's eye E.

As may be seen from the foregoing discussion it is possible to trackcertain user facial orientation characteristics using just a camera.However, many alternative forms of facial orientation characteristictracking setups could also be used. FIGS. 10A-10E illustrate examples offive facial orientation characteristic tracking systems that, amongother possible systems, can be implemented according to embodiments ofthe present invention.

In FIG. 10A, the user 1001 is facing a camera 1005 and infrared lightsensor 1007, which are mounted on top of a visual display 1003. To trackthe user's head tilt angle, the camera 1005 may be configured to performobject segmentation (i.e., track user's separate body parts) and thenestimate the user's head tilt angle from the information obtained. Thecamera 1005 and infrared light sensor 1007 are coupled to a processor1013 running software 1013, which may be configured as described above.By way of example, and not by way of limitation, object segmentation maybe accomplished using a motion model to describe how the image of atarget might change in accordance to different possible movements of theobject. It is noted that embodiments of the present invention may usemore than one camera, for example, some implementations may use twocameras. One camera can provide a zoomed-out image of the field of viewto locate the user, and a second camera can zoom-in and focus on theuser's face to provide a close-up image for better head and gazedirection estimation.

A user's eye gaze direction may also be acquired using this setup. Byway of example, and not by way of limitation, infrared light may beinitially directed towards the user's eyes from the infrared lightsensor 1007 and the reflection captured by the camera 1005. Theinformation extracted from the reflected infrared light will allow aprocessor coupled to the camera 1005 to determine an amount of eyerotation for the user. Video based eye trackers typically use thecorneal reflection and the center of the pupil as features to track overtime.

Thus, FIG. 10A illustrates a facial orientation characteristic trackingsetup that is configured to track both the user's head tilt angle andeye gaze direction in accordance with an embodiment of the presentinvention. It is noted that, for the purposes of example, it has beenassumed that the user is straight across from the display and camera.However, embodiments of the invention can be implemented even if theuser is not straight across from the display 1003 and/or camera 1005.For example, the user 1001 can be +45° or −45° to the right/left ofdisplay. As long as the user 1001 is within field of view of the camera205, the head angle θ_(H) and eye gaze θ_(E) can be estimated. Then, anormalized angle can be computed as a function of the location of user1001 with respect to the display 1003 and/or camera 1005 (e.g. bodyangle θ_(B) as shown in FIG. 10A), the head angle θ_(H) and eye gazeθ_(E). By way of example and not by way of limitation, if the user 1001,is located such that the body angle θ_(B) is +45° and if the head isturned at an angle θ_(H) of −45°, the user 1001 is fixing the deviationof the body from the display 1003 by turning his head, and this isalmost as good as having the person looking straight at the display.Specifically, if, e.g., the user's gaze angle θ_(E) is zero (i.e., theuser's pupils are centered), the normalized angle (e.g.,θ_(B)+θ_(H)+θ_(E)) is zero.

FIG. 10B provides another facial orientation characteristic trackingsetup. In FIG. 10B, the user 1001 is facing a camera 1005 mounted on topof a visual display 1003. The user 1001 is simultaneously wearing a pairof glasses 1009 (e.g., a pair of 3D shutter glasses) with a pair ofspaced-apart infrared (IR) light sources 1011 (e.g., one IR LED on eachlens of the glasses 1009). The camera 1005 may be configured to capturethe infrared light emanating from the light sources 1011, and thentriangulate user's head tilt angle from the information obtained.Because the position of the light sources 1011 will not varysignificantly with respect to its position on the user's face, thissetup will provide a relatively accurate estimation of the user's headtilt angle.

The glasses 1009 may additionally include a camera 1010 which canprovide images to the processor 1013 that can be used in conjunctionwith the software 1012 to find the location of the visual display 1003or to estimate the size of the visual display 203. By way of example,and not by way of limitation, the visual display be of a known typehaving known vertical and horizontal screen dimensions. A test image ofa known size relative to the screen may be displayed. Images of the testimage may be obtained by the camera and analyzed to determine theorientation and dimensions of the test image in the images obtained bythe camera 1010. Gathering this information allows the system tonormalize the user's facial orientation characteristic data so thatcalculation of those characteristics is independent of both the absolutelocations of the display 1003 and the user 1001. Moreover, the additionof the camera will allow the system to more accurately estimate visiblerange. Thus, FIG. 2B illustrates an alternative setup for determining auser's head tilt angle according to an aspect of the present disclosure.In some embodiments, separate cameras may be mounted to each lens of theglasses 1009 facing toward the user's eyes to facilitate gaze trackingby obtaining images of the eyes showing the relative location of thepupil with respect to the centers or corners of the eyes, e.g., asdiscussed above. The relatively fixed position of the glasses 1009relative to the user's eyes facilitates tracking the user's eye gazeangle θ_(E) independent of tracking of the user's head orientationθ_(H).

FIG. 10C provides a third facial orientation characteristic trackingsetup. In FIG. 10C, the user 1001 is facing a camera 1005 mounted on topof a visual display 1003. The user is also holding a controller 1015with one or more cameras 1017 (e.g., one on each side) configured tofacilitate interaction between the user 1001 and the contents on thevisual display 1003.

Images from the camera 1017 may be analyzed to determine the location ofthe visual display 1003 or to estimate the size of the visual display1003, e.g., using a displayed test image as in the above example.Gathering this information allows the system to normalize the user'sfacial orientation characteristic data so that calculation of thosecharacteristics is independent of both the absolute locations of thedisplay 1003 and the user 1001. Moreover, the addition of the cameras1017 to the controller 1015 allows the system to more accuratelyestimate visible range.

It is important to note that the setup in FIG. 10C may be furthercombined with the setup in FIG. 10A (not shown in FIG. 10C) in order totrack the user's eye gaze direction in addition to tracking the user'shead tilt angle while making the system independent of display size andlocation. Because the user's eyes are unobstructed in this setup, hiseye gaze direction may be obtained through the infrared light reflectionand capturing process discussed above.

FIG. 10D provides yet another alternative facial orientationcharacteristic tracking setup. In FIG. 10D, the user 1001 is facing acamera 1005 mounted on top of a visual display 1003. The user 1001 isalso wearing a headset 1019 with infrared light sources 1021 (e.g., oneon each eyepiece) and a microphone 1023, the headset 1019 beingconfigured to facilitate interaction between the user 1001 and thecontents on the visual display 1003. Much like the setup in FIG. 10B,the camera 1005 may capture images of the infrared light emanating fromthe light sources 1021 on the headset 1019, and the user's head tiltangle may be triangulated from analysis the images obtained. Because theposition of the headset 1019 tends not to vary significantly withrespect to its position on the user's face, this setup can provide arelatively accurate estimation of the user's head tilt angle.

In addition to tracking the user's head tilt angle using the infraredlight sensors 1021, the position of the user's head with respect to aspecified target may also be tracked by a separate microphone array 1027that is not part of the headset 1019. The microphone array 1027 may beconfigured to facilitate determination of a magnitude and orientation ofthe user's speech, e.g., using suitably configured software 1012 runningon the processor 1013. Examples of such methods are described e.g., incommonly assigned U.S. Pat. No. 7,783,061, commonly assigned U.S. Pat.No. 7,809,145, and commonly-assigned U.S. Patent Application Publicationnumber 2006/0239471, the entire contents of all three of which areincorporated herein by reference.

A detailed explanation of directional tracking of a user's speech usingthermographic information may be found in U.S. patent application Ser.No. 12/889,347, to Ruxin Chen and Steven Osman filed Sep. 23, 2010entitled “BLOW TRACKING USER INTERFACE SYSTEM AND METHOD”, which isherein incorporated by reference. By way of example, and not by way oflimitation, the orientation of the user's speech can be determined usinga thermal imaging camera to detect vibration patterns in the air aroundthe user's mouth that correspond to the sounds of the user's voiceduring speech. A time evolution of the vibration patterns can beanalyzed to determine a vector corresponding to a generalized directionof the user's speech.

Using both the position of the microphone array 1027 with respect to thecamera 1005 and the direction of the user's speech with respect to themicrophone array 1027, the position of the user's head with respect to aspecified target (e.g., display) may be calculated. To achieve greateraccuracy in establishing a user's head tilt angle, the infraredreflection and directional tracking methods for determining head tiltangle may be combined. Alternative embodiments may additionally includean inertial sensor 1027, as described with respect to FIG. 1A above.

The headset 1019 may additionally include a camera 1025 configured toobtain images of the visual display 1003 that may be analyzed to findthe location of the display and/or to estimate the size of the visualdisplay 1003. Gathering this information allows the system to normalizethe user's facial orientation characteristic data so that calculation ofthose characteristics is independent of both the absolute locations ofthe display 1003 and the user 1001. Moreover, the addition of the camerawill allow the system to more accurately estimate visible range. In someembodiments, one or more cameras 1025 may be mounted to the headset 1019facing toward the user's eyes to facilitate gaze tracking by obtainingimages of the eyes showing the relative location of the pupil withrespect to the centers or corners of the eyes, e.g., as discussed above.The relatively fixed position of the headset 1019 (and therefore, thecamera(s) 1025) relative to the user's eyes facilitates tracking theuser's eye gaze angle θ_(E) independent of tracking of the user's headorientation θ_(H).

It is important to note that the setup in FIG. 10D may be combined withthe setup in FIG. 10A (not shown in FIG. 10D) in order to track theuser's eye gaze direction in addition to tracking the user's head tiltangle. Because the user's eyes are unobstructed in this setup, his eyegaze direction may be obtained through infrared light reflection andcapturing process discussed above.

Embodiments of the present invention can also be implemented inhand-held devices, such as cell phones, tablet computers, personaldigital assistants, portable internet devices, or portable game devices,among other examples. FIG. 10E illustrates one possible example ofdetermining eye gaze direction in the context of a hand-held device1030. The device 1030 generally includes a processor 1039 which can beprogrammed with suitable software, e.g., as described above. The device1030 may include a display screen 1031 and camera 1035 coupled to theprocessor 1039. One or more microphones 1033 and control switches 1037may also be optionally coupled the processor 1039. The microphone 1033may be part of a microphone array. The control switches 1037 can be ofany type normally used with the particular type of hand-held device. Forexample, if the device 1030 is a cell phone, the control switches 237may include a numeric keypad or alpha-numeric keypad, touch screen, ortouch pad, as commonly used in such devices. Alternatively, if thedevice 1030 is a portable game unit, the control switches 1037 mayinclude digital or analog joysticks, digital control switches, triggers,and the like. In some embodiments, the display screen 1031 may be atouch screen interface and the functions of the control switches 1037may be implemented by the touch screen in conjunction with suitablesoftware, hardware or firmware. The camera 1035 may be configured toface the user 1001 when the user looks at the display screen 1031. Theprocessor 1039 may be programmed with software to implement head posetracking and/or eye-gaze tracking. The processor may be furtherconfigured to utilize head pose tracking and/or eye-gaze trackinginformation, e.g., as discussed above.

It is noted that the display screen 1031, microphone(s) 1033, camera1035, control switches 1037 and processor 1039 may be mounted to a casethat can be easily held in a user's hand or hands. In some embodiments,the device 1030 may operate in conjunction with a pair of specializedglasses, which may have features in common with the glasses 1009 shownin FIG. 10B and described hereinabove. Such glasses may communicate withthe processor through a wireless or wired connection, e.g., a personalarea network connection, such as a Bluetooth network connection. In someembodiments, the device 1030 may be used in conjunction with a headset,which can have features in common with the headset 1019 shown in FIG.10D and described hereinabove. Such a headset may communicate with theprocessor through a wireless or wired connection, e.g., a personal areanetwork connection, such as a Bluetooth network connection. The device1030 may include suitable antenna and transceiver to facilitate wirelessnetwork connection.

It is noted that the examples depicted in FIGS. 10A-10E are only a fewexamples of many setups that could be used to track a user's facialorientation characteristics in accordance with aspects of the presentdisclosure. Similarly, various body and other facial orientationcharacteristics in addition to the head tilt angle and eye gazedirection described above may be tracked to facilitate the adjustment ofregions of interest rendered on a display.

ADDITIONAL ASPECTS

Aspects of the present disclosure also include a non-transitorycomputer-readable medium having computer executable instructionsembodied therein that, when executed, implement graphics processing inaccordance with the above-mentioned aspects, e.g., as described abovewith respect to FIG. 1A, FIG. 2, FIG. 4A, FIG. 4B, FIG. 5A, FIG. 5B, andFIGS. 6A-6D. Examples of suitable computer-readable media include, butare not limited to RAM, ROM, hard disk, flash drive, SDRAM, CD-ROM,Blu-Ray, magnetic tape, and floppy disk.

CONCLUSION

It is noted that aspects of the present disclosure have been describedwith reference to eye tracking devices that use infrared light sources,which has developed as a relatively standard light source for opticaleye tracking techniques. However, it is understood that otherimplementations are possible. For example, in implementations of thepresent disclosure, other invisible light sources are possible, such asultraviolet light. By way of further example, in implementations of thepresent disclosure, visible light sources are possible for eyeillumination, although it may be desirable to use invisible lightsources in order to avoid distracting a user.

While the above is a complete description of the preferred embodiment ofthe present invention, it is possible to use various alternatives,modifications and equivalents. Therefore, the scope of the presentinvention should be determined not with reference to the abovedescription but should, instead, be determined with reference to theappended claims, along with their full scope of equivalents. Any featuredescribed herein, whether preferred or not, may be combined with anyother feature described herein, whether preferred or not. In the claimsthat follow, the indefinite article “a”, or “an” refers to a quantity ofone or more of the item following the article, except where expresslystated otherwise. The appended claims are not to be interpreted asincluding means-plus-function limitations, unless such a limitation isexplicitly recited in a given claim using the phrase “means for.”

The invention claimed is:
 1. A method comprising: obtaining gazetracking data representing a user's gaze with respect to one or moreimages transmitted to a user; analyzing the gaze tracking data todetermine one or more regions of interest; and selectively compressingthe one or more images to reduce the number of bits needed to transmitdata corresponding to the one or more images to the user so that fewerbits are needed to transmit data for portions of an image outside theone or more regions interest than for portions of the image within theone or more regions of interest wherein selectively compressing the oneor more images includes ignoring packet loss for transmission of one ormore parts of the image outside the one or more regions of interest ofthe one or more transmitted images.
 2. The method of claim 1, whereinselectively compressing the one or more images includes selectivelycompressing the image so that the one or, more regions of interestundergo less compression than portions of the image outside the one ormore regions of interest.
 3. The method of claim 1, wherein selectivelycompressing the one or more images includes using a higher quantizationparameter for portions of the image that are outside the regions ofinterest than for portions of the image within the one or more regionsof interest.
 4. The method of claim 1, further comprising reducingtransmission power for transmitting the one or more transmitted imagesfor a period of time during which the user blinks or the user's visionundergoes a saccade.
 5. The method of claim 1, further comprisingceasing transmission of the one or more images for time period duringwhich the user blinks.
 6. The method of claim 1, further comprisingceasing transmission of the one or more images for a time period duringwhich a user's vision undergoes a saccade.
 7. The method of claim 1,wherein said analyzing the gaze tracking data includes analyzing arotational velocity and/or rotational acceleration of the user's eye. 8.The method of claim 1, wherein said analyzing the gaze tracking dataincludes analyzing information related to movement of the user's eyelid.9. The method of claim 1, wherein said analyzing the gaze tracking datato determine the onset and duration of a vision tracking event includesanalyzing previous gaze tracking data to improve detection and estimatesof duration estimation for vision interrupting events.
 10. The method ofclaim 9, further comprising determining one or more gaze trackingparameters from the gaze tracking data; generating adjusted foveationdata representing an adjusted size and/or shape of one or more regionsof interest in one or more images to be subsequently presented to theuser based on the one or more gaze tracking parameters; generatingfoveated image data representing one or more foveated images using theadjusted foveation data, wherein the one or more foveated images arecharacterized by level of detail within the one or more regions ofinterest and lower level of detail outside the one or more regions ofinterest; and presenting the one or more foveated images with thefoveated image data to the user.
 11. The method of claim 10, whereindetermining one or more gaze tracking error parameters from the gazetracking data includes determining a rate of rotation of a user's eyewith respect to one or more axes.
 12. The method of claim 10, whereindetermining one or more gaze tracking error parameters from the gazetracking data includes determining a rate of rotation of a user's eyewith respect to one or more axes and determining an error in a fixationof the user's eye on a region of interest of the one or more regions ofinterest.
 13. The method of claim 10, wherein determining one or moregaze tracking error parameters from the gaze tracking data includesdetermining a rate of rotation of a user's eye with respect to one ormore axes and determining whether the user's eye is moving in smoothpursuit.
 14. The method of claim 10, wherein determining one or moregaze tracking error parameters from the gaze tracking data includesdetermining a confidence interval regarding the current gaze position.15. The method of claim 10, wherein, determining one or more gazetracking error parameters from the gaze tracking data includesdetermining the blink cycle of a user.
 16. The method of claim 10,wherein determining one or more gaze tracking error parameters from thegaze tracking data includes determining the saccade cycle of a user. 17.The method of claim 10, wherein determining one or more gaze trackingerror parameters from the gaze tracking data includes determining atransition in the gaze direction of a user as a result of a change inthe depth of field of the presented image.
 18. The method of claim 10,wherein determining one or more gaze tracking error parameters from thegaze tracking data includes determining whether a user is color blind.19. The method of claim 10, wherein determining one or more gazetracking error parameters from the gaze tracking data includesdetermining a rate of rotation of a user's eye with respect to one ormore axes and determining the level of gaze stability of the user's eye.20. The method of claim 10, wherein determining one or more gazetracking error parameters from the gaze tracking data includesdetermining a rate of rotation of a user's eye with respect to one ormore axes and determining whether the user's eye movement is a precursorto the user's head movement.
 21. The method of claim 10, wherein theadjusted foveation data includes data characterizing a particular regionof interest of the one or more regions of interest, wherein the datacharacterizing the particular region of interest includes a foveationregion and a transition region, wherein the foveation region ischaracterized by a higher level of detail than the transition region andthe transition region is characterized by a higher level of detail thanregions of a corresponding image outside the particular region ofinterest.
 22. The method of claim 21, wherein generating adjustedfoveation data representing an adjusted size and/or shape of one or moreregions of interest includes adjusting a size of the transition region.23. The method of claim 10, wherein the adjusted foveation data includesdata representing vertex density in the one or more regions of interest.24. The method of claim 10, wherein the adjusted foveation data includesdata representing pixel resolution in the one or more regions ofinterest.
 25. The method of claim 10, further comprising creating astandard 2D compliant image from the foveated image data forpresentation on one or more additional displays.
 26. The method of claim1, further comprising reducing transmission power for transmitting theone or more transmitted images for a period of time during which theuser blinks or the user's vision undergoes a saccade and subsequentlyincreasing transmission power.
 27. The method of claim 1, furthercomprising generating the one or more transmitted images by generatingfoveated image data representing one or more foveated images usingfoveation data representing one or more regions of interest of the imagedetermined from the gaze tracking data, wherein the one or more foveatedimages are characterized by level of detail within the one or moreregions of interest and lower level of detail outside the one or moreregions of interest.
 28. The method of claim 27, wherein the foveationdata includes data representing vertex density in the one or moreregions of interest.
 29. The method of claim 27, wherein the foveationdata includes data representing pixel resolution in the one or moreregions of interest.
 30. The method of claim 27, further comprisingcreating a standard 2D compliant image from the foveated image data forpresentation on one or more additional displays.
 31. The method of claim1, wherein selectively compressing the one or more images includesomitting error packet transmission for portions of the one or moreimages outside the one or more regions interest but not for portions ofthe one or more images within the one or more regions of interest.
 32. Asystem, comprising: a processor; a memory; and computer-readableinstructions embodied in the memory, the computer-readable instructionsbeing configured to implement a method when executed, the methodcomprising: obtaining gaze tracking data representing a user's gaze withrespect to one or more images transmitted to a user; analyzing the gazetracking data to determine one or more regions of interest; andselectively compressing the one or more images to reduce the number ofbits needed to transmit data corresponding to the one or more images tothe user so that fewer bits are needed to transmit data for portions ofan image outside the one or more regions interest than for portions ofthe image within the one or more regions of interest wherein selectivelycompressing the one or more images includes ignoring packet loss fortransmission of one or more parts of the image outside the one or moreregions of interest of the one or more transmitted images.
 33. Anon-transitory computer-readable medium having computer-readableinstructions embodied therein, the computer-readable instructions beingconfigured to implement a method when executed, the method comprising:obtaining gaze tracking data representing a user's gaze with respect toone or more images transmitted to a user; analyzing the gaze trackingdata to determine one or more, regions of interest; and selectivelycompressing the one or more images to reduce the number of bits neededto transmit data corresponding to the one or more images to the user sothat fewer bits are needed to transmit data for portions of an imageoutside the one or more regions interest than for portions of the imagewithin the one or more regions of interest wherein selectivelycompressing the one or more images includes ignoring packet loss fortransmission of one or more parts of the image outside the one or moreregions of interest of the one or more transmitted images.